API reference¶
parsel.csstranslator¶
-
class
parsel.csstranslator.
GenericTranslator
[source]¶ Bases:
parsel.csstranslator.TranslatorMixin
,cssselect.xpath.GenericTranslator
-
css_to_xpath
(**kwds)[source]¶ Translate a group of selectors to XPath.
Pseudo-elements are not supported here since XPath only knows about “real” elements.
Parameters: - css – A group of selectors as an Unicode string.
- prefix – This string is prepended to the XPath expression for each selector. The default makes selectors scoped to the context node’s subtree.
Raises: SelectorSyntaxError
on invalid selectors,ExpressionError
on unknown/unsupported selectors, including pseudo-elements.Returns: The equivalent XPath 1.0 expression as an Unicode string.
-
-
class
parsel.csstranslator.
HTMLTranslator
(xhtml=False)[source]¶ Bases:
parsel.csstranslator.TranslatorMixin
,cssselect.xpath.HTMLTranslator
-
css_to_xpath
(**kwds)[source]¶ Translate a group of selectors to XPath.
Pseudo-elements are not supported here since XPath only knows about “real” elements.
Parameters: - css – A group of selectors as an Unicode string.
- prefix – This string is prepended to the XPath expression for each selector. The default makes selectors scoped to the context node’s subtree.
Raises: SelectorSyntaxError
on invalid selectors,ExpressionError
on unknown/unsupported selectors, including pseudo-elements.Returns: The equivalent XPath 1.0 expression as an Unicode string.
-
-
class
parsel.csstranslator.
TranslatorMixin
[source]¶ Bases:
object
This mixin adds support to CSS pseudo elements via dynamic dispatch.
Currently supported pseudo-elements are
::text
and::attr(ATTR_NAME)
.-
xpath_attr_functional_pseudo_element
(xpath, function)[source]¶ Support selecting attribute values using ::attr() pseudo-element
-
parsel.selector¶
XPath selectors based on lxml
-
class
parsel.selector.
Selector
(text=None, type=None, namespaces=None, root=None, base_url=None, _expr=None)[source]¶ Bases:
object
Selector
allows you to select parts of an XML or HTML text using CSS or XPath expressions and extract data from it.text
is aunicode
object in Python 2 or astr
object in Python 3type
defines the selector type, it can be"html"
,"xml"
orNone
(default). Iftype
isNone
, the selector defaults to"html"
.base_url
allows setting a URL for the document. This is needed when looking up external entities with relative paths. See [lxml documentation](https://lxml.de/api/index.html)lxml.etree.fromstring
for more information.-
attrib
¶ Return the attributes dictionary for underlying element.
-
css
(query)[source]¶ Apply the given CSS selector and return a
SelectorList
instance.query
is a string containing the CSS selector to apply.In the background, CSS queries are translated into XPath queries using cssselect library and run
.xpath()
method.
-
extract
()¶ Serialize and return the matched nodes in a single unicode string. Percent encoded content is unquoted.
-
get
()[source]¶ Serialize and return the matched nodes in a single unicode string. Percent encoded content is unquoted.
-
namespaces
¶
-
re
(regex, replace_entities=True)[source]¶ Apply the given regex and return a list of unicode strings with the matches.
regex
can be either a compiled regular expression or a string which will be compiled to a regular expression usingre.compile(regex)
.By default, character entity references are replaced by their corresponding character (except for
&
and<
). Passingreplace_entities
asFalse
switches off these replacements.
-
re_first
(regex, default=None, replace_entities=True)[source]¶ Apply the given regex and return the first unicode string which matches. If there is no match, return the default value (
None
if the argument is not provided).By default, character entity references are replaced by their corresponding character (except for
&
and<
). Passingreplace_entities
asFalse
switches off these replacements.
-
register_namespace
(prefix, uri)[source]¶ Register the given namespace to be used in this
Selector
. Without registering namespaces you can’t select or extract data from non-standard namespaces. See Working on XML (and namespaces).
-
remove_namespaces
()[source]¶ Remove all namespaces, allowing to traverse the document using namespace-less xpaths. See Removing namespaces.
-
root
¶
-
selectorlist_cls
¶ alias of
SelectorList
-
text
¶
-
type
¶
-
xpath
(query, namespaces=None, **kwargs)[source]¶ Find nodes matching the xpath
query
and return the result as aSelectorList
instance with all elements flattened. List elements implementSelector
interface too.query
is a string containing the XPATH query to apply.namespaces
is an optionalprefix: namespace-uri
mapping (dict) for additional prefixes to those registered withregister_namespace(prefix, uri)
. Contrary toregister_namespace()
, these prefixes are not saved for future calls.Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:
selector.xpath('//a[href=$url]', url="http://www.example.com")
-
-
class
parsel.selector.
SelectorList
[source]¶ Bases:
list
The
SelectorList
class is a subclass of the builtinlist
class, which provides a few additional methods.-
attrib
¶ Return the attributes dictionary for the first element. If the list is empty, return an empty dict.
-
css
(query)[source]¶ Call the
.css()
method for each element in this list and return their results flattened as anotherSelectorList
.query
is the same argument as the one inSelector.css()
-
extract
()¶ Call the
.get()
method for each element is this list and return their results flattened, as a list of unicode strings.
-
extract_first
(default=None)¶ Return the result of
.get()
for the first element in this list. If the list is empty, return the default value.
-
get
(default=None)[source]¶ Return the result of
.get()
for the first element in this list. If the list is empty, return the default value.
-
getall
()[source]¶ Call the
.get()
method for each element is this list and return their results flattened, as a list of unicode strings.
-
re
(regex, replace_entities=True)[source]¶ Call the
.re()
method for each element in this list and return their results flattened, as a list of unicode strings.By default, character entity references are replaced by their corresponding character (except for
&
and<
. Passingreplace_entities
asFalse
switches off these replacements.
-
re_first
(regex, default=None, replace_entities=True)[source]¶ Call the
.re()
method for the first element in this list and return the result in an unicode string. If the list is empty or the regex doesn’t match anything, return the default value (None
if the argument is not provided).By default, character entity references are replaced by their corresponding character (except for
&
and<
. Passingreplace_entities
asFalse
switches off these replacements.
-
xpath
(xpath, namespaces=None, **kwargs)[source]¶ Call the
.xpath()
method for each element in this list and return their results flattened as anotherSelectorList
.query
is the same argument as the one inSelector.xpath()
namespaces
is an optionalprefix: namespace-uri
mapping (dict) for additional prefixes to those registered withregister_namespace(prefix, uri)
. Contrary toregister_namespace()
, these prefixes are not saved for future calls.Any additional named arguments can be used to pass values for XPath variables in the XPath expression, e.g.:
selector.xpath('//a[href=$url]', url="http://www.example.com")
-
parsel.utils¶
-
parsel.utils.
extract_regex
(regex, text, replace_entities=True)[source]¶ Extract a list of unicode strings from the given text/encoding using the following policies: * if the regex contains a named group called “extract” that will be returned * if the regex contains multiple numbered groups, all those will be returned (flattened) * if the regex doesn’t contain any group the entire regex matching is returned
-
parsel.utils.
flatten
(sequence) → list[source]¶ Returns a single, flat list which contains all elements retrieved from the sequence and all recursively contained sub-sequences (iterables). Examples: >>> [1, 2, [3,4], (5,6)] [1, 2, [3, 4], (5, 6)] >>> flatten([[[1,2,3], (42,None)], [4,5], [6], 7, (8,9,10)]) [1, 2, 3, 42, None, 4, 5, 6, 7, 8, 9, 10] >>> flatten([“foo”, “bar”]) [‘foo’, ‘bar’] >>> flatten([“foo”, [“baz”, 42], “bar”]) [‘foo’, ‘baz’, 42, ‘bar’]