diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..8dba2df
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,6 @@
+/config/*.tmplc
+*.pyc
+*.pyo
+/_cache
+/_output
+/.venv
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 0000000..fe55838
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,32 @@
+FROM ubuntu:bionic
+
+RUN apt-get update \
+ && apt-get install -y --no-install-recommends \
+ python-pip python-setuptools python-wheel \
+ locales tzdata \
+ ca-certificates \
+ strace gdb lsof locate net-tools htop iputils-ping dnsutils \
+ python2.7-dbg python2.7 libpython2.7 python-dbg libpython-dbg \
+ curl nano vim tree less telnet patch \
+ graphviz sqlite3 \
+ dumb-init \
+ && rm -rf /var/lib/apt/lists/*
+
+RUN locale-gen en_US.UTF-8
+
+COPY /code /planet/code
+COPY /config /planet/config
+COPY /static /planet/dist/static
+
+WORKDIR /planet
+
+ENTRYPOINT ["dumb-init"]
+
+RUN echo "#!/bin/bash -eux \n\
+python2.7 code/planet.py config/config.ini \n\
+cd /planet/dist/ \n\
+python2.7 -mSimpleHTTPServer 8080 \n\
+"> /start.sh
+RUN chmod +x /start.sh
+EXPOSE 8080
+
diff --git a/LICENSE b/LICENSE
deleted file mode 100644
index 9cecc1d..0000000
--- a/LICENSE
+++ /dev/null
@@ -1,674 +0,0 @@
- GNU GENERAL PUBLIC LICENSE
- Version 3, 29 June 2007
-
- Copyright (C) 2007 Free Software Foundation, Inc.
- Everyone is permitted to copy and distribute verbatim copies
- of this license document, but changing it is not allowed.
-
- Preamble
-
- The GNU General Public License is a free, copyleft license for
-software and other kinds of works.
-
- The licenses for most software and other practical works are designed
-to take away your freedom to share and change the works. By contrast,
-the GNU General Public License is intended to guarantee your freedom to
-share and change all versions of a program--to make sure it remains free
-software for all its users. We, the Free Software Foundation, use the
-GNU General Public License for most of our software; it applies also to
-any other work released this way by its authors. You can apply it to
-your programs, too.
-
- When we speak of free software, we are referring to freedom, not
-price. Our General Public Licenses are designed to make sure that you
-have the freedom to distribute copies of free software (and charge for
-them if you wish), that you receive source code or can get it if you
-want it, that you can change the software or use pieces of it in new
-free programs, and that you know you can do these things.
-
- To protect your rights, we need to prevent others from denying you
-these rights or asking you to surrender the rights. Therefore, you have
-certain responsibilities if you distribute copies of the software, or if
-you modify it: responsibilities to respect the freedom of others.
-
- For example, if you distribute copies of such a program, whether
-gratis or for a fee, you must pass on to the recipients the same
-freedoms that you received. You must make sure that they, too, receive
-or can get the source code. And you must show them these terms so they
-know their rights.
-
- Developers that use the GNU GPL protect your rights with two steps:
-(1) assert copyright on the software, and (2) offer you this License
-giving you legal permission to copy, distribute and/or modify it.
-
- For the developers' and authors' protection, the GPL clearly explains
-that there is no warranty for this free software. For both users' and
-authors' sake, the GPL requires that modified versions be marked as
-changed, so that their problems will not be attributed erroneously to
-authors of previous versions.
-
- Some devices are designed to deny users access to install or run
-modified versions of the software inside them, although the manufacturer
-can do so. This is fundamentally incompatible with the aim of
-protecting users' freedom to change the software. The systematic
-pattern of such abuse occurs in the area of products for individuals to
-use, which is precisely where it is most unacceptable. Therefore, we
-have designed this version of the GPL to prohibit the practice for those
-products. If such problems arise substantially in other domains, we
-stand ready to extend this provision to those domains in future versions
-of the GPL, as needed to protect the freedom of users.
-
- Finally, every program is threatened constantly by software patents.
-States should not allow patents to restrict development and use of
-software on general-purpose computers, but in those that do, we wish to
-avoid the special danger that patents applied to a free program could
-make it effectively proprietary. To prevent this, the GPL assures that
-patents cannot be used to render the program non-free.
-
- The precise terms and conditions for copying, distribution and
-modification follow.
-
- TERMS AND CONDITIONS
-
- 0. Definitions.
-
- "This License" refers to version 3 of the GNU General Public License.
-
- "Copyright" also means copyright-like laws that apply to other kinds of
-works, such as semiconductor masks.
-
- "The Program" refers to any copyrightable work licensed under this
-License. Each licensee is addressed as "you". "Licensees" and
-"recipients" may be individuals or organizations.
-
- To "modify" a work means to copy from or adapt all or part of the work
-in a fashion requiring copyright permission, other than the making of an
-exact copy. The resulting work is called a "modified version" of the
-earlier work or a work "based on" the earlier work.
-
- A "covered work" means either the unmodified Program or a work based
-on the Program.
-
- To "propagate" a work means to do anything with it that, without
-permission, would make you directly or secondarily liable for
-infringement under applicable copyright law, except executing it on a
-computer or modifying a private copy. Propagation includes copying,
-distribution (with or without modification), making available to the
-public, and in some countries other activities as well.
-
- To "convey" a work means any kind of propagation that enables other
-parties to make or receive copies. Mere interaction with a user through
-a computer network, with no transfer of a copy, is not conveying.
-
- An interactive user interface displays "Appropriate Legal Notices"
-to the extent that it includes a convenient and prominently visible
-feature that (1) displays an appropriate copyright notice, and (2)
-tells the user that there is no warranty for the work (except to the
-extent that warranties are provided), that licensees may convey the
-work under this License, and how to view a copy of this License. If
-the interface presents a list of user commands or options, such as a
-menu, a prominent item in the list meets this criterion.
-
- 1. Source Code.
-
- The "source code" for a work means the preferred form of the work
-for making modifications to it. "Object code" means any non-source
-form of a work.
-
- A "Standard Interface" means an interface that either is an official
-standard defined by a recognized standards body, or, in the case of
-interfaces specified for a particular programming language, one that
-is widely used among developers working in that language.
-
- The "System Libraries" of an executable work include anything, other
-than the work as a whole, that (a) is included in the normal form of
-packaging a Major Component, but which is not part of that Major
-Component, and (b) serves only to enable use of the work with that
-Major Component, or to implement a Standard Interface for which an
-implementation is available to the public in source code form. A
-"Major Component", in this context, means a major essential component
-(kernel, window system, and so on) of the specific operating system
-(if any) on which the executable work runs, or a compiler used to
-produce the work, or an object code interpreter used to run it.
-
- The "Corresponding Source" for a work in object code form means all
-the source code needed to generate, install, and (for an executable
-work) run the object code and to modify the work, including scripts to
-control those activities. However, it does not include the work's
-System Libraries, or general-purpose tools or generally available free
-programs which are used unmodified in performing those activities but
-which are not part of the work. For example, Corresponding Source
-includes interface definition files associated with source files for
-the work, and the source code for shared libraries and dynamically
-linked subprograms that the work is specifically designed to require,
-such as by intimate data communication or control flow between those
-subprograms and other parts of the work.
-
- The Corresponding Source need not include anything that users
-can regenerate automatically from other parts of the Corresponding
-Source.
-
- The Corresponding Source for a work in source code form is that
-same work.
-
- 2. Basic Permissions.
-
- All rights granted under this License are granted for the term of
-copyright on the Program, and are irrevocable provided the stated
-conditions are met. This License explicitly affirms your unlimited
-permission to run the unmodified Program. The output from running a
-covered work is covered by this License only if the output, given its
-content, constitutes a covered work. This License acknowledges your
-rights of fair use or other equivalent, as provided by copyright law.
-
- You may make, run and propagate covered works that you do not
-convey, without conditions so long as your license otherwise remains
-in force. You may convey covered works to others for the sole purpose
-of having them make modifications exclusively for you, or provide you
-with facilities for running those works, provided that you comply with
-the terms of this License in conveying all material for which you do
-not control copyright. Those thus making or running the covered works
-for you must do so exclusively on your behalf, under your direction
-and control, on terms that prohibit them from making any copies of
-your copyrighted material outside their relationship with you.
-
- Conveying under any other circumstances is permitted solely under
-the conditions stated below. Sublicensing is not allowed; section 10
-makes it unnecessary.
-
- 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
-
- No covered work shall be deemed part of an effective technological
-measure under any applicable law fulfilling obligations under article
-11 of the WIPO copyright treaty adopted on 20 December 1996, or
-similar laws prohibiting or restricting circumvention of such
-measures.
-
- When you convey a covered work, you waive any legal power to forbid
-circumvention of technological measures to the extent such circumvention
-is effected by exercising rights under this License with respect to
-the covered work, and you disclaim any intention to limit operation or
-modification of the work as a means of enforcing, against the work's
-users, your or third parties' legal rights to forbid circumvention of
-technological measures.
-
- 4. Conveying Verbatim Copies.
-
- You may convey verbatim copies of the Program's source code as you
-receive it, in any medium, provided that you conspicuously and
-appropriately publish on each copy an appropriate copyright notice;
-keep intact all notices stating that this License and any
-non-permissive terms added in accord with section 7 apply to the code;
-keep intact all notices of the absence of any warranty; and give all
-recipients a copy of this License along with the Program.
-
- You may charge any price or no price for each copy that you convey,
-and you may offer support or warranty protection for a fee.
-
- 5. Conveying Modified Source Versions.
-
- You may convey a work based on the Program, or the modifications to
-produce it from the Program, in the form of source code under the
-terms of section 4, provided that you also meet all of these conditions:
-
- a) The work must carry prominent notices stating that you modified
- it, and giving a relevant date.
-
- b) The work must carry prominent notices stating that it is
- released under this License and any conditions added under section
- 7. This requirement modifies the requirement in section 4 to
- "keep intact all notices".
-
- c) You must license the entire work, as a whole, under this
- License to anyone who comes into possession of a copy. This
- License will therefore apply, along with any applicable section 7
- additional terms, to the whole of the work, and all its parts,
- regardless of how they are packaged. This License gives no
- permission to license the work in any other way, but it does not
- invalidate such permission if you have separately received it.
-
- d) If the work has interactive user interfaces, each must display
- Appropriate Legal Notices; however, if the Program has interactive
- interfaces that do not display Appropriate Legal Notices, your
- work need not make them do so.
-
- A compilation of a covered work with other separate and independent
-works, which are not by their nature extensions of the covered work,
-and which are not combined with it such as to form a larger program,
-in or on a volume of a storage or distribution medium, is called an
-"aggregate" if the compilation and its resulting copyright are not
-used to limit the access or legal rights of the compilation's users
-beyond what the individual works permit. Inclusion of a covered work
-in an aggregate does not cause this License to apply to the other
-parts of the aggregate.
-
- 6. Conveying Non-Source Forms.
-
- You may convey a covered work in object code form under the terms
-of sections 4 and 5, provided that you also convey the
-machine-readable Corresponding Source under the terms of this License,
-in one of these ways:
-
- a) Convey the object code in, or embodied in, a physical product
- (including a physical distribution medium), accompanied by the
- Corresponding Source fixed on a durable physical medium
- customarily used for software interchange.
-
- b) Convey the object code in, or embodied in, a physical product
- (including a physical distribution medium), accompanied by a
- written offer, valid for at least three years and valid for as
- long as you offer spare parts or customer support for that product
- model, to give anyone who possesses the object code either (1) a
- copy of the Corresponding Source for all the software in the
- product that is covered by this License, on a durable physical
- medium customarily used for software interchange, for a price no
- more than your reasonable cost of physically performing this
- conveying of source, or (2) access to copy the
- Corresponding Source from a network server at no charge.
-
- c) Convey individual copies of the object code with a copy of the
- written offer to provide the Corresponding Source. This
- alternative is allowed only occasionally and noncommercially, and
- only if you received the object code with such an offer, in accord
- with subsection 6b.
-
- d) Convey the object code by offering access from a designated
- place (gratis or for a charge), and offer equivalent access to the
- Corresponding Source in the same way through the same place at no
- further charge. You need not require recipients to copy the
- Corresponding Source along with the object code. If the place to
- copy the object code is a network server, the Corresponding Source
- may be on a different server (operated by you or a third party)
- that supports equivalent copying facilities, provided you maintain
- clear directions next to the object code saying where to find the
- Corresponding Source. Regardless of what server hosts the
- Corresponding Source, you remain obligated to ensure that it is
- available for as long as needed to satisfy these requirements.
-
- e) Convey the object code using peer-to-peer transmission, provided
- you inform other peers where the object code and Corresponding
- Source of the work are being offered to the general public at no
- charge under subsection 6d.
-
- A separable portion of the object code, whose source code is excluded
-from the Corresponding Source as a System Library, need not be
-included in conveying the object code work.
-
- A "User Product" is either (1) a "consumer product", which means any
-tangible personal property which is normally used for personal, family,
-or household purposes, or (2) anything designed or sold for incorporation
-into a dwelling. In determining whether a product is a consumer product,
-doubtful cases shall be resolved in favor of coverage. For a particular
-product received by a particular user, "normally used" refers to a
-typical or common use of that class of product, regardless of the status
-of the particular user or of the way in which the particular user
-actually uses, or expects or is expected to use, the product. A product
-is a consumer product regardless of whether the product has substantial
-commercial, industrial or non-consumer uses, unless such uses represent
-the only significant mode of use of the product.
-
- "Installation Information" for a User Product means any methods,
-procedures, authorization keys, or other information required to install
-and execute modified versions of a covered work in that User Product from
-a modified version of its Corresponding Source. The information must
-suffice to ensure that the continued functioning of the modified object
-code is in no case prevented or interfered with solely because
-modification has been made.
-
- If you convey an object code work under this section in, or with, or
-specifically for use in, a User Product, and the conveying occurs as
-part of a transaction in which the right of possession and use of the
-User Product is transferred to the recipient in perpetuity or for a
-fixed term (regardless of how the transaction is characterized), the
-Corresponding Source conveyed under this section must be accompanied
-by the Installation Information. But this requirement does not apply
-if neither you nor any third party retains the ability to install
-modified object code on the User Product (for example, the work has
-been installed in ROM).
-
- The requirement to provide Installation Information does not include a
-requirement to continue to provide support service, warranty, or updates
-for a work that has been modified or installed by the recipient, or for
-the User Product in which it has been modified or installed. Access to a
-network may be denied when the modification itself materially and
-adversely affects the operation of the network or violates the rules and
-protocols for communication across the network.
-
- Corresponding Source conveyed, and Installation Information provided,
-in accord with this section must be in a format that is publicly
-documented (and with an implementation available to the public in
-source code form), and must require no special password or key for
-unpacking, reading or copying.
-
- 7. Additional Terms.
-
- "Additional permissions" are terms that supplement the terms of this
-License by making exceptions from one or more of its conditions.
-Additional permissions that are applicable to the entire Program shall
-be treated as though they were included in this License, to the extent
-that they are valid under applicable law. If additional permissions
-apply only to part of the Program, that part may be used separately
-under those permissions, but the entire Program remains governed by
-this License without regard to the additional permissions.
-
- When you convey a copy of a covered work, you may at your option
-remove any additional permissions from that copy, or from any part of
-it. (Additional permissions may be written to require their own
-removal in certain cases when you modify the work.) You may place
-additional permissions on material, added by you to a covered work,
-for which you have or can give appropriate copyright permission.
-
- Notwithstanding any other provision of this License, for material you
-add to a covered work, you may (if authorized by the copyright holders of
-that material) supplement the terms of this License with terms:
-
- a) Disclaiming warranty or limiting liability differently from the
- terms of sections 15 and 16 of this License; or
-
- b) Requiring preservation of specified reasonable legal notices or
- author attributions in that material or in the Appropriate Legal
- Notices displayed by works containing it; or
-
- c) Prohibiting misrepresentation of the origin of that material, or
- requiring that modified versions of such material be marked in
- reasonable ways as different from the original version; or
-
- d) Limiting the use for publicity purposes of names of licensors or
- authors of the material; or
-
- e) Declining to grant rights under trademark law for use of some
- trade names, trademarks, or service marks; or
-
- f) Requiring indemnification of licensors and authors of that
- material by anyone who conveys the material (or modified versions of
- it) with contractual assumptions of liability to the recipient, for
- any liability that these contractual assumptions directly impose on
- those licensors and authors.
-
- All other non-permissive additional terms are considered "further
-restrictions" within the meaning of section 10. If the Program as you
-received it, or any part of it, contains a notice stating that it is
-governed by this License along with a term that is a further
-restriction, you may remove that term. If a license document contains
-a further restriction but permits relicensing or conveying under this
-License, you may add to a covered work material governed by the terms
-of that license document, provided that the further restriction does
-not survive such relicensing or conveying.
-
- If you add terms to a covered work in accord with this section, you
-must place, in the relevant source files, a statement of the
-additional terms that apply to those files, or a notice indicating
-where to find the applicable terms.
-
- Additional terms, permissive or non-permissive, may be stated in the
-form of a separately written license, or stated as exceptions;
-the above requirements apply either way.
-
- 8. Termination.
-
- You may not propagate or modify a covered work except as expressly
-provided under this License. Any attempt otherwise to propagate or
-modify it is void, and will automatically terminate your rights under
-this License (including any patent licenses granted under the third
-paragraph of section 11).
-
- However, if you cease all violation of this License, then your
-license from a particular copyright holder is reinstated (a)
-provisionally, unless and until the copyright holder explicitly and
-finally terminates your license, and (b) permanently, if the copyright
-holder fails to notify you of the violation by some reasonable means
-prior to 60 days after the cessation.
-
- Moreover, your license from a particular copyright holder is
-reinstated permanently if the copyright holder notifies you of the
-violation by some reasonable means, this is the first time you have
-received notice of violation of this License (for any work) from that
-copyright holder, and you cure the violation prior to 30 days after
-your receipt of the notice.
-
- Termination of your rights under this section does not terminate the
-licenses of parties who have received copies or rights from you under
-this License. If your rights have been terminated and not permanently
-reinstated, you do not qualify to receive new licenses for the same
-material under section 10.
-
- 9. Acceptance Not Required for Having Copies.
-
- You are not required to accept this License in order to receive or
-run a copy of the Program. Ancillary propagation of a covered work
-occurring solely as a consequence of using peer-to-peer transmission
-to receive a copy likewise does not require acceptance. However,
-nothing other than this License grants you permission to propagate or
-modify any covered work. These actions infringe copyright if you do
-not accept this License. Therefore, by modifying or propagating a
-covered work, you indicate your acceptance of this License to do so.
-
- 10. Automatic Licensing of Downstream Recipients.
-
- Each time you convey a covered work, the recipient automatically
-receives a license from the original licensors, to run, modify and
-propagate that work, subject to this License. You are not responsible
-for enforcing compliance by third parties with this License.
-
- An "entity transaction" is a transaction transferring control of an
-organization, or substantially all assets of one, or subdividing an
-organization, or merging organizations. If propagation of a covered
-work results from an entity transaction, each party to that
-transaction who receives a copy of the work also receives whatever
-licenses to the work the party's predecessor in interest had or could
-give under the previous paragraph, plus a right to possession of the
-Corresponding Source of the work from the predecessor in interest, if
-the predecessor has it or can get it with reasonable efforts.
-
- You may not impose any further restrictions on the exercise of the
-rights granted or affirmed under this License. For example, you may
-not impose a license fee, royalty, or other charge for exercise of
-rights granted under this License, and you may not initiate litigation
-(including a cross-claim or counterclaim in a lawsuit) alleging that
-any patent claim is infringed by making, using, selling, offering for
-sale, or importing the Program or any portion of it.
-
- 11. Patents.
-
- A "contributor" is a copyright holder who authorizes use under this
-License of the Program or a work on which the Program is based. The
-work thus licensed is called the contributor's "contributor version".
-
- A contributor's "essential patent claims" are all patent claims
-owned or controlled by the contributor, whether already acquired or
-hereafter acquired, that would be infringed by some manner, permitted
-by this License, of making, using, or selling its contributor version,
-but do not include claims that would be infringed only as a
-consequence of further modification of the contributor version. For
-purposes of this definition, "control" includes the right to grant
-patent sublicenses in a manner consistent with the requirements of
-this License.
-
- Each contributor grants you a non-exclusive, worldwide, royalty-free
-patent license under the contributor's essential patent claims, to
-make, use, sell, offer for sale, import and otherwise run, modify and
-propagate the contents of its contributor version.
-
- In the following three paragraphs, a "patent license" is any express
-agreement or commitment, however denominated, not to enforce a patent
-(such as an express permission to practice a patent or covenant not to
-sue for patent infringement). To "grant" such a patent license to a
-party means to make such an agreement or commitment not to enforce a
-patent against the party.
-
- If you convey a covered work, knowingly relying on a patent license,
-and the Corresponding Source of the work is not available for anyone
-to copy, free of charge and under the terms of this License, through a
-publicly available network server or other readily accessible means,
-then you must either (1) cause the Corresponding Source to be so
-available, or (2) arrange to deprive yourself of the benefit of the
-patent license for this particular work, or (3) arrange, in a manner
-consistent with the requirements of this License, to extend the patent
-license to downstream recipients. "Knowingly relying" means you have
-actual knowledge that, but for the patent license, your conveying the
-covered work in a country, or your recipient's use of the covered work
-in a country, would infringe one or more identifiable patents in that
-country that you have reason to believe are valid.
-
- If, pursuant to or in connection with a single transaction or
-arrangement, you convey, or propagate by procuring conveyance of, a
-covered work, and grant a patent license to some of the parties
-receiving the covered work authorizing them to use, propagate, modify
-or convey a specific copy of the covered work, then the patent license
-you grant is automatically extended to all recipients of the covered
-work and works based on it.
-
- A patent license is "discriminatory" if it does not include within
-the scope of its coverage, prohibits the exercise of, or is
-conditioned on the non-exercise of one or more of the rights that are
-specifically granted under this License. You may not convey a covered
-work if you are a party to an arrangement with a third party that is
-in the business of distributing software, under which you make payment
-to the third party based on the extent of your activity of conveying
-the work, and under which the third party grants, to any of the
-parties who would receive the covered work from you, a discriminatory
-patent license (a) in connection with copies of the covered work
-conveyed by you (or copies made from those copies), or (b) primarily
-for and in connection with specific products or compilations that
-contain the covered work, unless you entered into that arrangement,
-or that patent license was granted, prior to 28 March 2007.
-
- Nothing in this License shall be construed as excluding or limiting
-any implied license or other defenses to infringement that may
-otherwise be available to you under applicable patent law.
-
- 12. No Surrender of Others' Freedom.
-
- If conditions are imposed on you (whether by court order, agreement or
-otherwise) that contradict the conditions of this License, they do not
-excuse you from the conditions of this License. If you cannot convey a
-covered work so as to satisfy simultaneously your obligations under this
-License and any other pertinent obligations, then as a consequence you may
-not convey it at all. For example, if you agree to terms that obligate you
-to collect a royalty for further conveying from those to whom you convey
-the Program, the only way you could satisfy both those terms and this
-License would be to refrain entirely from conveying the Program.
-
- 13. Use with the GNU Affero General Public License.
-
- Notwithstanding any other provision of this License, you have
-permission to link or combine any covered work with a work licensed
-under version 3 of the GNU Affero General Public License into a single
-combined work, and to convey the resulting work. The terms of this
-License will continue to apply to the part which is the covered work,
-but the special requirements of the GNU Affero General Public License,
-section 13, concerning interaction through a network will apply to the
-combination as such.
-
- 14. Revised Versions of this License.
-
- The Free Software Foundation may publish revised and/or new versions of
-the GNU General Public License from time to time. Such new versions will
-be similar in spirit to the present version, but may differ in detail to
-address new problems or concerns.
-
- Each version is given a distinguishing version number. If the
-Program specifies that a certain numbered version of the GNU General
-Public License "or any later version" applies to it, you have the
-option of following the terms and conditions either of that numbered
-version or of any later version published by the Free Software
-Foundation. If the Program does not specify a version number of the
-GNU General Public License, you may choose any version ever published
-by the Free Software Foundation.
-
- If the Program specifies that a proxy can decide which future
-versions of the GNU General Public License can be used, that proxy's
-public statement of acceptance of a version permanently authorizes you
-to choose that version for the Program.
-
- Later license versions may give you additional or different
-permissions. However, no additional obligations are imposed on any
-author or copyright holder as a result of your choosing to follow a
-later version.
-
- 15. Disclaimer of Warranty.
-
- THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
-APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
-HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
-OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
-THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
-IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
-ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
-
- 16. Limitation of Liability.
-
- IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
-WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
-THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
-GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
-USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
-DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
-PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
-EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
-SUCH DAMAGES.
-
- 17. Interpretation of Sections 15 and 16.
-
- If the disclaimer of warranty and limitation of liability provided
-above cannot be given local legal effect according to their terms,
-reviewing courts shall apply local law that most closely approximates
-an absolute waiver of all civil liability in connection with the
-Program, unless a warranty or assumption of liability accompanies a
-copy of the Program in return for a fee.
-
- END OF TERMS AND CONDITIONS
-
- How to Apply These Terms to Your New Programs
-
- If you develop a new program, and you want it to be of the greatest
-possible use to the public, the best way to achieve this is to make it
-free software which everyone can redistribute and change under these terms.
-
- To do so, attach the following notices to the program. It is safest
-to attach them to the start of each source file to most effectively
-state the exclusion of warranty; and each file should have at least
-the "copyright" line and a pointer to where the full notice is found.
-
- {one line to give the program's name and a brief idea of what it does.}
- Copyright (C) {year} {name of author}
-
- This program is free software: you can redistribute it and/or modify
- it under the terms of the GNU General Public License as published by
- the Free Software Foundation, either version 3 of the License, or
- (at your option) any later version.
-
- This program is distributed in the hope that it will be useful,
- but WITHOUT ANY WARRANTY; without even the implied warranty of
- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- GNU General Public License for more details.
-
- You should have received a copy of the GNU General Public License
- along with this program. If not, see .
-
-Also add information on how to contact you by electronic and paper mail.
-
- If the program does terminal interaction, make it output a short
-notice like this when it starts in an interactive mode:
-
- {project} Copyright (C) {year} {fullname}
- This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
- This is free software, and you are welcome to redistribute it
- under certain conditions; type `show c' for details.
-
-The hypothetical commands `show w' and `show c' should show the appropriate
-parts of the General Public License. Of course, your program's commands
-might be different; for a GUI interface, you would use an "about box".
-
- You should also get your employer (if you work as a programmer) or school,
-if any, to sign a "copyright disclaimer" for the program, if necessary.
-For more information on this, and how to apply and follow the GNU GPL, see
-.
-
- The GNU General Public License does not permit incorporating your program
-into proprietary programs. If your program is a subroutine library, you
-may consider it more useful to permit linking proprietary applications with
-the library. If this is what you want to do, use the GNU Lesser General
-Public License instead of this License. But first, please read
-.
diff --git a/README.md b/README.md
index 844591f..1f3d15b 100644
--- a/README.md
+++ b/README.md
@@ -2,9 +2,11 @@
Repo público para atualização da lista de sites do https://planet.python.org.br/
+Recuperado do repo planet python https://github.com/python/planet
+
## Adição de Feeds:
-Para adicionar um feed, faça um PR com a adição no arquivo config/planet.conf
+Para adicionar um feed, faça um PR com a adição no arquivo config/config.ini
## Template
diff --git a/code/README.pydotorg b/code/README.pydotorg
new file mode 100644
index 0000000..f4dca19
--- /dev/null
+++ b/code/README.pydotorg
@@ -0,0 +1,5 @@
+
+This holds a copy of the PlanetPlanet code (www.planetplanet.org).
+This version is a copy of the nightly tarball, taken on 2005-05-15.
+
+--amk
diff --git a/code/TODO b/code/TODO
new file mode 100644
index 0000000..5b5f668
--- /dev/null
+++ b/code/TODO
@@ -0,0 +1,85 @@
+Feature Goals for Planet 1.0
+============================
+
+ * Store and use feed history
+
+ We have a huge problem at the moment, in that we only display the current
+ contents of the feeds we download. The old Netscape standard was 15 items
+ in a feed, so many people still use that. Unfortunately, on sites like
+ Slashdot, those 15 items move *very* quickly. So as they fall off the feed,
+ they fall off your Planet aggregate. That is bad, bad, bad, and we need to
+ solve it. Any ideas? There are potential bugs all over this. :-)
+
+ * Add multiple keyword support
+
+ I hadn't announced it yet, but I have added a simple 'keyword feeds'
+ feature, which allows you to subscribe to feeds and only display items that
+ mention your keyword in the title, summary or description. This is totally
+ arse-kickingly rad, and demonstrated on my personal Planet.
+
+ However, currently it only allows you to define one keyword phrase for each
+ feed, which kinda sucks. If there's some quick-and-dirty boolean phrase
+ parser, we could use it really well here, for instance:
+
+ keyword = (gnome OR kde) NOT sucks
+
+ Maybe that's overkill - we could just support multiple ORed keyword fields.
+
+ * Fix stupid UTF-8 error with current keyword support
+
+ For some reason, keyword support tweaks string encoding errors, such as:
+ "WARNING:root:Item wasn't in UTF-8 or ISO-8859-1, replaced all non-ASCII
+ characters." This is bad.
+
+ * Allow templates to use extra feed elements
+
+ See Frederic's thread about Freshmeat the other day. I don't want to
+ special case everything, and I don't want to stomp on item variables we
+ already have, but it would be nice to bring special things from feeds up
+ into the template data structure. Should we only do this for namespaced
+ elements (extensions)?
+
+ * Fix intermittent timezone detection / arithmetic bugs
+
+ From Nick Moffitt: "Right now there's something screwy in the way planet
+ deals with time zones, so that all you Australians end up claiming the top
+ of my planet while merkins I want to read end up shoved down a ways."
+
+ * Allow display normalisation to specified timezone
+
+ Some Planet admins would like their feed to be displayed in the local
+ timezone, instead of UTC.
+
+ * Add "sort by entry received date" option
+
+ From Nick Moffitt: "Is there a way to get planet to not do entry-date
+ placement at all? Like, always put new entries up in the order they were
+ first seen by planet? I use mine to implement the "show me stuff I haven't
+ already read yet" feature that pretty much all Web sites lack.
+
+ Answer from Scott J. Remnant: "Yeah, this would be actually pretty easy ...
+ it already has this code for entries missing times entirely, I'll add a
+ config option to just outright ignore feed times and sort by seen order.
+ I'll make it a per-feed option, maybe "sort_order = seen/time" and if you
+ want it global just stick it in [DEFAULT]."
+
+ * Support OPML and foaf subscriptions
+
+ This might be a bit invasive, but I want to be able to subscribe to OPML
+ and FOAF files, and see each feed as if it were subscribed individually.
+ Perhaps we can do this with a two-pass configuration scheme, first to pull
+ the static configs, second to go fetch and generate the dynamic configs.
+ The more I think about it, the less invasive it sounds. Hmm.
+
+ * Provide a 'disabled' configuration option
+
+ Make it easy to mark a feed as disabled, but still show it on the subs
+ list. Perhaps disabled is the wrong word. Some people want to use Planet to
+ do their blogroll management, but not actually show some feeds in the
+ aggregation. If someone can think of a better word, please tell me.
+ Perhaps 'hide'.
+
+Other Feature Goals
+===================
+
+ * Port to feedparser 3.0
diff --git a/code/planet-cache.py b/code/planet-cache.py
new file mode 100755
index 0000000..9334583
--- /dev/null
+++ b/code/planet-cache.py
@@ -0,0 +1,194 @@
+#!/usr/bin/env python
+# -*- coding: UTF-8 -*-
+"""Planet cache tool.
+
+"""
+
+__authors__ = [ "Scott James Remnant ",
+ "Jeff Waugh " ]
+__license__ = "Python"
+
+
+import os
+import sys
+import time
+import dbhash
+import ConfigParser
+
+import planet
+
+
+def usage():
+ print "Usage: planet-cache [options] CACHEFILE [ITEMID]..."
+ print
+ print "Examine and modify information in the Planet cache."
+ print
+ print "Channel Commands:"
+ print " -C, --channel Display known information on the channel"
+ print " -L, --list List items in the channel"
+ print " -K, --keys List all keys found in channel items"
+ print
+ print "Item Commands (need ITEMID):"
+ print " -I, --item Display known information about the item(s)"
+ print " -H, --hide Mark the item(s) as hidden"
+ print " -U, --unhide Mark the item(s) as not hidden"
+ print
+ print "Other Options:"
+ print " -h, --help Display this help message and exit"
+ sys.exit(0)
+
+def usage_error(msg, *args):
+ print >>sys.stderr, msg, " ".join(args)
+ print >>sys.stderr, "Perhaps you need --help ?"
+ sys.exit(1)
+
+def print_keys(item, title):
+ keys = item.keys()
+ keys.sort()
+ key_len = max([ len(k) for k in keys ])
+
+ print title + ":"
+ for key in keys:
+ if item.key_type(key) == item.DATE:
+ value = time.strftime(planet.TIMEFMT_ISO, item[key])
+ else:
+ value = str(item[key])
+ print " %-*s %s" % (key_len, key, fit_str(value, 74 - key_len))
+
+def fit_str(string, length):
+ if len(string) <= length:
+ return string
+ else:
+ return string[:length-4] + " ..."
+
+
+if __name__ == "__main__":
+ cache_file = None
+ want_ids = 0
+ ids = []
+
+ command = None
+
+ for arg in sys.argv[1:]:
+ if arg == "-h" or arg == "--help":
+ usage()
+ elif arg == "-C" or arg == "--channel":
+ if command is not None:
+ usage_error("Only one command option may be supplied")
+ command = "channel"
+ elif arg == "-L" or arg == "--list":
+ if command is not None:
+ usage_error("Only one command option may be supplied")
+ command = "list"
+ elif arg == "-K" or arg == "--keys":
+ if command is not None:
+ usage_error("Only one command option may be supplied")
+ command = "keys"
+ elif arg == "-I" or arg == "--item":
+ if command is not None:
+ usage_error("Only one command option may be supplied")
+ command = "item"
+ want_ids = 1
+ elif arg == "-H" or arg == "--hide":
+ if command is not None:
+ usage_error("Only one command option may be supplied")
+ command = "hide"
+ want_ids = 1
+ elif arg == "-U" or arg == "--unhide":
+ if command is not None:
+ usage_error("Only one command option may be supplied")
+ command = "unhide"
+ want_ids = 1
+ elif arg.startswith("-"):
+ usage_error("Unknown option:", arg)
+ else:
+ if cache_file is None:
+ cache_file = arg
+ elif want_ids:
+ ids.append(arg)
+ else:
+ usage_error("Unexpected extra argument:", arg)
+
+ if cache_file is None:
+ usage_error("Missing expected cache filename")
+ elif want_ids and not len(ids):
+ usage_error("Missing expected entry ids")
+
+ # Open the cache file directly to get the URL it represents
+ try:
+ db = dbhash.open(cache_file)
+ url = db["url"]
+ db.close()
+ except dbhash.bsddb._db.DBError, e:
+ print >>sys.stderr, cache_file + ":", e.args[1]
+ sys.exit(1)
+ except KeyError:
+ print >>sys.stderr, cache_file + ": Probably not a cache file"
+ sys.exit(1)
+
+ # Now do it the right way :-)
+ my_planet = planet.Planet(ConfigParser.ConfigParser())
+ my_planet.cache_directory = os.path.dirname(cache_file)
+ channel = planet.Channel(my_planet, url)
+
+ for item_id in ids:
+ if not channel.has_item(item_id):
+ print >>sys.stderr, item_id + ": Not in channel"
+ sys.exit(1)
+
+ # Do the user's bidding
+ if command == "channel":
+ print_keys(channel, "Channel Keys")
+
+ elif command == "item":
+ for item_id in ids:
+ item = channel.get_item(item_id)
+ print_keys(item, "Item Keys for %s" % item_id)
+
+ elif command == "list":
+ print "Items in Channel:"
+ for item in channel.items(hidden=1, sorted=1):
+ print " " + item.id
+ print " " + time.strftime(planet.TIMEFMT_ISO, item.date)
+ if hasattr(item, "title"):
+ print " " + fit_str(item.title, 70)
+ if hasattr(item, "hidden"):
+ print " (hidden)"
+
+ elif command == "keys":
+ keys = {}
+ for item in channel.items():
+ for key in item.keys():
+ keys[key] = 1
+
+ keys = keys.keys()
+ keys.sort()
+
+ print "Keys used in Channel:"
+ for key in keys:
+ print " " + key
+ print
+
+ print "Use --item to output values of particular items."
+
+ elif command == "hide":
+ for item_id in ids:
+ item = channel.get_item(item_id)
+ if hasattr(item, "hidden"):
+ print item_id + ": Already hidden."
+ else:
+ item.hidden = "yes"
+
+ channel.cache_write()
+ print "Done."
+
+ elif command == "unhide":
+ for item_id in ids:
+ item = channel.get_item(item_id)
+ if hasattr(item, "hidden"):
+ del(item.hidden)
+ else:
+ print item_id + ": Not hidden."
+
+ channel.cache_write()
+ print "Done."
diff --git a/code/planet.py b/code/planet.py
new file mode 100755
index 0000000..41141b6
--- /dev/null
+++ b/code/planet.py
@@ -0,0 +1,154 @@
+#!/usr/bin/env python
+"""The Planet aggregator.
+
+A flexible and easy-to-use aggregator for generating websites.
+
+Visit http://www.planetplanet.org/ for more information and to download
+the latest version.
+
+Requires Python 2.1, recommends 2.3.
+"""
+
+__authors__ = [ "Scott James Remnant ",
+ "Jeff Waugh " ]
+__license__ = "Python"
+
+
+import os
+import sys
+import time
+import locale
+import socket
+import urlparse
+
+import planet
+
+from ConfigParser import ConfigParser
+
+# Default configuration file path
+CONFIG_FILE = "config.ini"
+
+# Defaults for the [Planet] config section
+PLANET_NAME = "Unconfigured Planet"
+PLANET_LINK = "Unconfigured Planet"
+PLANET_FEED = None
+OWNER_NAME = "Anonymous Coward"
+OWNER_EMAIL = ""
+LOG_LEVEL = "WARNING"
+FEED_TIMEOUT = 20 # seconds
+
+# Default template file list
+TEMPLATE_FILES = "examples/basic/planet.html.tmpl"
+
+
+
+def config_get(config, section, option, default=None, raw=0, vars=None):
+ """Get a value from the configuration, with a default."""
+ if config.has_option(section, option):
+ return config.get(section, option, raw=raw, vars=None)
+ else:
+ return default
+
+def main():
+ config_file = CONFIG_FILE
+ offline = 0
+ verbose = 0
+
+ for arg in sys.argv[1:]:
+ if arg == "-h" or arg == "--help":
+ print "Usage: planet [options] [CONFIGFILE]"
+ print
+ print "Options:"
+ print " -v, --verbose DEBUG level logging during update"
+ print " -o, --offline Update the Planet from the cache only"
+ print " -h, --help Display this help message and exit"
+ print
+ sys.exit(0)
+ elif arg == "-v" or arg == "--verbose":
+ verbose = 1
+ elif arg == "-o" or arg == "--offline":
+ offline = 1
+ elif arg.startswith("-"):
+ print >>sys.stderr, "Unknown option:", arg
+ sys.exit(1)
+ else:
+ config_file = arg
+
+ # Read the configuration file
+ config = ConfigParser()
+ config.read(config_file)
+ if not config.has_section("Planet"):
+ print >>sys.stderr, "Configuration missing [Planet] section."
+ sys.exit(1)
+
+ # Read the [Planet] config section
+ planet_name = config_get(config, "Planet", "name", PLANET_NAME)
+ planet_link = config_get(config, "Planet", "link", PLANET_LINK)
+ planet_feed = config_get(config, "Planet", "feed", PLANET_FEED)
+ owner_name = config_get(config, "Planet", "owner_name", OWNER_NAME)
+ owner_email = config_get(config, "Planet", "owner_email", OWNER_EMAIL)
+ if verbose:
+ log_level = "DEBUG"
+ else:
+ log_level = config_get(config, "Planet", "log_level", LOG_LEVEL)
+ feed_timeout = config_get(config, "Planet", "feed_timeout", FEED_TIMEOUT)
+ template_files = config_get(config, "Planet", "template_files",
+ TEMPLATE_FILES).split(" ")
+
+ # Default feed to the first feed for which there is a template
+ if not planet_feed:
+ for template_file in template_files:
+ name = os.path.splitext(os.path.basename(template_file))[0]
+ if name.find('atom')>=0 or name.find('rss')>=0:
+ planet_feed = urlparse.urljoin(planet_link, name)
+ break
+
+ # Define locale
+ if config.has_option("Planet", "locale"):
+ # The user can specify more than one locale (separated by ":") as
+ # fallbacks.
+ locale_ok = False
+ for user_locale in config.get("Planet", "locale").split(':'):
+ user_locale = user_locale.strip()
+ try:
+ locale.setlocale(locale.LC_ALL, user_locale)
+ except locale.Error:
+ pass
+ else:
+ locale_ok = True
+ break
+ if not locale_ok:
+ print >>sys.stderr, "Unsupported locale setting."
+ sys.exit(1)
+
+ # Activate logging
+ planet.logging.basicConfig()
+ planet.logging.getLogger().setLevel(planet.logging.getLevelName(log_level))
+ log = planet.logging.getLogger("planet.runner")
+ try:
+ log.warning
+ except:
+ log.warning = log.warn
+
+ if feed_timeout:
+ try:
+ feed_timeout = float(feed_timeout)
+ except:
+ log.warning("Feed timeout set to invalid value '%s', skipping", feed_timeout)
+ feed_timeout = None
+
+ if feed_timeout and not offline:
+ socket.setdefaulttimeout(feed_timeout)
+ log.debug("Socket timeout set to %d seconds", feed_timeout)
+
+ # run the planet
+ my_planet = planet.Planet(config)
+ my_planet.run(planet_name, planet_link, template_files, offline)
+
+ my_planet.generate_all_files(template_files, planet_name,
+ planet_link, planet_feed, owner_name, owner_email)
+
+
+if __name__ == "__main__":
+ main()
+
diff --git a/code/planet/__init__.py b/code/planet/__init__.py
new file mode 100644
index 0000000..929920b
--- /dev/null
+++ b/code/planet/__init__.py
@@ -0,0 +1,953 @@
+#!/usr/bin/env python
+# -*- coding: UTF-8 -*-
+"""Planet aggregator library.
+
+This package is a library for developing web sites or software that
+aggregate RSS, CDF and Atom feeds taken from elsewhere into a single,
+combined feed.
+"""
+
+__version__ = "2.0"
+__authors__ = [ "Scott James Remnant ",
+ "Jeff Waugh " ]
+__license__ = "Python"
+
+
+# Modules available without separate import
+import cache
+import feedparser
+import sanitize
+import htmltmpl
+import sgmllib
+try:
+ import logging
+except:
+ import compat_logging as logging
+
+# Limit the effect of "from planet import *"
+__all__ = ("cache", "feedparser", "htmltmpl", "logging",
+ "Planet", "Channel", "NewsItem")
+
+
+import os
+import md5
+import time
+import dbhash
+import re
+
+try:
+ from xml.sax.saxutils import escape
+except:
+ def escape(data):
+ return data.replace("&","&").replace(">",">").replace("<","<")
+
+# Version information (for generator headers)
+VERSION = ("Planet/%s +http://www.planetplanet.org" % __version__)
+
+# Default User-Agent header to send when retreiving feeds
+USER_AGENT = VERSION + " " + feedparser.USER_AGENT
+
+# Default cache directory
+CACHE_DIRECTORY = "cache"
+
+# Default number of items to display from a new feed
+NEW_FEED_ITEMS = 10
+
+# Useful common date/time formats
+TIMEFMT_ISO = "%Y-%m-%dT%H:%M:%S+00:00"
+TIMEFMT_822 = "%a, %d %b %Y %H:%M:%S +0000"
+
+
+# Log instance to use here
+log = logging.getLogger("planet")
+try:
+ log.warning
+except:
+ log.warning = log.warn
+
+# Defaults for the template file config sections
+ENCODING = "utf-8"
+ITEMS_PER_PAGE = 60
+DAYS_PER_PAGE = 0
+OUTPUT_DIR = "output"
+DATE_FORMAT = "%B %d, %Y %I:%M %p"
+NEW_DATE_FORMAT = "%B %d, %Y"
+ACTIVITY_THRESHOLD = 0
+
+class stripHtml(sgmllib.SGMLParser):
+ "remove all tags from the data"
+ def __init__(self, data):
+ sgmllib.SGMLParser.__init__(self)
+ self.result=''
+ self.feed(data)
+ self.close()
+ def handle_data(self, data):
+ if data: self.result+=data
+
+def template_info(item, date_format):
+ """Produce a dictionary of template information."""
+ info = {}
+ for key in item.keys():
+ if item.key_type(key) == item.DATE:
+ date = item.get_as_date(key)
+ info[key] = time.strftime(date_format, date)
+ info[key + "_iso"] = time.strftime(TIMEFMT_ISO, date)
+ info[key + "_822"] = time.strftime(TIMEFMT_822, date)
+ else:
+ info[key] = item[key]
+ if 'title' in item.keys():
+ info['title_plain'] = stripHtml(info['title']).result
+
+ return info
+
+
+class Planet:
+ """A set of channels.
+
+ This class represents a set of channels for which the items will
+ be aggregated together into one combined feed.
+
+ Properties:
+ user_agent User-Agent header to fetch feeds with.
+ cache_directory Directory to store cached channels in.
+ new_feed_items Number of items to display from a new feed.
+ filter A regular expression that articles must match.
+ exclude A regular expression that articles must not match.
+ """
+ def __init__(self, config):
+ self.config = config
+
+ self._channels = []
+
+ self.user_agent = USER_AGENT
+ self.cache_directory = CACHE_DIRECTORY
+ self.new_feed_items = NEW_FEED_ITEMS
+ self.filter = None
+ self.exclude = None
+
+ def tmpl_config_get(self, template, option, default=None, raw=0, vars=None):
+ """Get a template value from the configuration, with a default."""
+ if self.config.has_option(template, option):
+ return self.config.get(template, option, raw=raw, vars=None)
+ elif self.config.has_option("Planet", option):
+ return self.config.get("Planet", option, raw=raw, vars=None)
+ else:
+ return default
+
+ def gather_channel_info(self, template_file="Planet"):
+ date_format = self.tmpl_config_get(template_file,
+ "date_format", DATE_FORMAT, raw=1)
+
+ activity_threshold = int(self.tmpl_config_get(template_file,
+ "activity_threshold",
+ ACTIVITY_THRESHOLD))
+
+ if activity_threshold:
+ activity_horizon = \
+ time.gmtime(time.time()-86400*activity_threshold)
+ else:
+ activity_horizon = 0
+
+ channels = {}
+ channels_list = []
+ for channel in self.channels(hidden=1):
+ channels[channel] = template_info(channel, date_format)
+ channels_list.append(channels[channel])
+
+ # identify inactive feeds
+ if activity_horizon:
+ latest = channel.items(sorted=1)
+ if len(latest)==0 or latest[0].date < activity_horizon:
+ channels[channel]["message"] = \
+ "no activity in %d days" % activity_threshold
+
+ # report channel level errors
+ if not channel.url_status: continue
+ status = int(channel.url_status)
+ if status == 403:
+ channels[channel]["message"] = "403: forbidden"
+ elif status == 404:
+ channels[channel]["message"] = "404: not found"
+ elif status == 408:
+ channels[channel]["message"] = "408: request timeout"
+ elif status == 410:
+ channels[channel]["message"] = "410: gone"
+ elif status == 500:
+ channels[channel]["message"] = "internal server error"
+ elif status >= 400:
+ channels[channel]["message"] = "http status %s" % status
+
+ return channels, channels_list
+
+ def gather_items_info(self, channels, template_file="Planet", channel_list=None):
+ items_list = []
+ prev_date = []
+ prev_channel = None
+
+ date_format = self.tmpl_config_get(template_file,
+ "date_format", DATE_FORMAT, raw=1)
+ items_per_page = int(self.tmpl_config_get(template_file,
+ "items_per_page", ITEMS_PER_PAGE))
+ days_per_page = int(self.tmpl_config_get(template_file,
+ "days_per_page", DAYS_PER_PAGE))
+ new_date_format = self.tmpl_config_get(template_file,
+ "new_date_format", NEW_DATE_FORMAT, raw=1)
+
+ for newsitem in self.items(max_items=items_per_page,
+ max_days=days_per_page,
+ channels=channel_list):
+ item_info = template_info(newsitem, date_format)
+ chan_info = channels[newsitem._channel]
+ for k, v in chan_info.items():
+ item_info["channel_" + k] = v
+
+ # Check for the start of a new day
+ if prev_date[:3] != newsitem.date[:3]:
+ prev_date = newsitem.date
+ item_info["new_date"] = time.strftime(new_date_format,
+ newsitem.date)
+
+ # Check for the start of a new channel
+ if item_info.has_key("new_date") \
+ or prev_channel != newsitem._channel:
+ prev_channel = newsitem._channel
+ item_info["new_channel"] = newsitem._channel.url
+
+ items_list.append(item_info)
+
+ return items_list
+
+ def run(self, planet_name, planet_link, template_files, offline = False):
+ log = logging.getLogger("planet.runner")
+
+ # Create a planet
+ log.info("Loading cached data")
+ if self.config.has_option("Planet", "cache_directory"):
+ self.cache_directory = self.config.get("Planet", "cache_directory")
+ if self.config.has_option("Planet", "new_feed_items"):
+ self.new_feed_items = int(self.config.get("Planet", "new_feed_items"))
+ self.user_agent = "%s +%s %s" % (planet_name, planet_link,
+ self.user_agent)
+ if self.config.has_option("Planet", "filter"):
+ self.filter = self.config.get("Planet", "filter")
+
+ # The other configuration blocks are channels to subscribe to
+ for feed_url in self.config.sections():
+ if feed_url == "Planet" or feed_url in template_files:
+ continue
+
+ # Create a channel, configure it and subscribe it
+ channel = Channel(self, feed_url)
+ self.subscribe(channel)
+
+ # Update it
+ try:
+ if not offline and not channel.url_status == '410':
+ channel.update()
+ except KeyboardInterrupt:
+ raise
+ except:
+ log.exception("Update of <%s> failed", feed_url)
+
+ def generate_all_files(self, template_files, planet_name,
+ planet_link, planet_feed, owner_name, owner_email):
+
+ log = logging.getLogger("planet.runner")
+ # Go-go-gadget-template
+ for template_file in template_files:
+ manager = htmltmpl.TemplateManager()
+ log.info("Processing template %s", template_file)
+ try:
+ template = manager.prepare(template_file)
+ except htmltmpl.TemplateError:
+ template = manager.prepare(os.path.basename(template_file))
+ # Read the configuration
+ output_dir = self.tmpl_config_get(template_file,
+ "output_dir", OUTPUT_DIR)
+ date_format = self.tmpl_config_get(template_file,
+ "date_format", DATE_FORMAT, raw=1)
+ encoding = self.tmpl_config_get(template_file, "encoding", ENCODING)
+
+ # We treat each template individually
+ base = os.path.splitext(os.path.basename(template_file))[0]
+ url = os.path.join(planet_link, base)
+ output_file = os.path.join(output_dir, base)
+
+ # Gather information
+ channels, channels_list = self.gather_channel_info(template_file)
+ items_list = self.gather_items_info(channels, template_file)
+
+ # Gather item information
+
+ # Process the template
+ tp = htmltmpl.TemplateProcessor(html_escape=0)
+ tp.set("Items", items_list)
+ tp.set("Channels", channels_list)
+
+ # Generic information
+ tp.set("generator", VERSION)
+ tp.set("name", planet_name)
+ tp.set("link", planet_link)
+ tp.set("owner_name", owner_name)
+ tp.set("owner_email", owner_email)
+ tp.set("url", url)
+
+ if planet_feed:
+ tp.set("feed", planet_feed)
+ tp.set("feedtype", planet_feed.find('rss')>=0 and 'rss' or 'atom')
+
+ # Update time
+ date = time.gmtime()
+ tp.set("date", time.strftime(date_format, date))
+ tp.set("date_iso", time.strftime(TIMEFMT_ISO, date))
+ tp.set("date_822", time.strftime(TIMEFMT_822, date))
+
+ try:
+ log.info("Writing %s", output_file)
+ output_fd = open(output_file, "w")
+ if encoding.lower() in ("utf-8", "utf8"):
+ # UTF-8 output is the default because we use that internally
+ output_fd.write(tp.process(template))
+ elif encoding.lower() in ("xml", "html", "sgml"):
+ # Magic for Python 2.3 users
+ output = tp.process(template).decode("utf-8")
+ output_fd.write(output.encode("ascii", "xmlcharrefreplace"))
+ else:
+ # Must be a "known" encoding
+ output = tp.process(template).decode("utf-8")
+ output_fd.write(output.encode(encoding, "replace"))
+ output_fd.close()
+ except KeyboardInterrupt:
+ raise
+ except:
+ log.exception("Write of %s failed", output_file)
+
+ def channels(self, hidden=0, sorted=1):
+ """Return the list of channels."""
+ channels = []
+ for channel in self._channels:
+ if hidden or not channel.has_key("hidden"):
+ channels.append((channel.name, channel))
+
+ if sorted:
+ channels.sort()
+
+ return [ c[-1] for c in channels ]
+
+ def find_by_basename(self, basename):
+ for channel in self._channels:
+ if basename == channel.cache_basename(): return channel
+
+ def subscribe(self, channel):
+ """Subscribe the planet to the channel."""
+ self._channels.append(channel)
+
+ def unsubscribe(self, channel):
+ """Unsubscribe the planet from the channel."""
+ self._channels.remove(channel)
+
+ def items(self, hidden=0, sorted=1, max_items=0, max_days=0, channels=None):
+ """Return an optionally filtered list of items in the channel.
+
+ The filters are applied in the following order:
+
+ If hidden is true then items in hidden channels and hidden items
+ will be returned.
+
+ If sorted is true then the item list will be sorted with the newest
+ first.
+
+ If max_items is non-zero then this number of items, at most, will
+ be returned.
+
+ If max_days is non-zero then any items older than the newest by
+ this number of days won't be returned. Requires sorted=1 to work.
+
+
+ The sharp-eyed will note that this looks a little strange code-wise,
+ it turns out that Python gets *really* slow if we try to sort the
+ actual items themselves. Also we use mktime here, but it's ok
+ because we discard the numbers and just need them to be relatively
+ consistent between each other.
+ """
+ planet_filter_re = None
+ if self.filter:
+ planet_filter_re = re.compile(self.filter, re.I)
+ planet_exclude_re = None
+ if self.exclude:
+ planet_exclude_re = re.compile(self.exclude, re.I)
+
+ items = []
+ seen_guids = {}
+ if not channels: channels=self.channels(hidden=hidden, sorted=0)
+ for channel in channels:
+ for item in channel._items.values():
+ if hidden or not item.has_key("hidden"):
+
+ channel_filter_re = None
+ if channel.filter:
+ channel_filter_re = re.compile(channel.filter,
+ re.I)
+ channel_exclude_re = None
+ if channel.exclude:
+ channel_exclude_re = re.compile(channel.exclude,
+ re.I)
+ if (planet_filter_re or planet_exclude_re \
+ or channel_filter_re or channel_exclude_re):
+ title = ""
+ if item.has_key("title"):
+ title = item.title
+ content = item.get_content("content")
+
+ if planet_filter_re:
+ if not (planet_filter_re.search(title) \
+ or planet_filter_re.search(content)):
+ continue
+
+ if planet_exclude_re:
+ if (planet_exclude_re.search(title) \
+ or planet_exclude_re.search(content)):
+ continue
+
+ if channel_filter_re:
+ if not (channel_filter_re.search(title) \
+ or channel_filter_re.search(content)):
+ continue
+
+ if channel_exclude_re:
+ if (channel_exclude_re.search(title) \
+ or channel_exclude_re.search(content)):
+ continue
+
+ if not seen_guids.has_key(item.id):
+ seen_guids[item.id] = 1;
+ items.append((time.mktime(item.date), item.order, item))
+
+ # Sort the list
+ if sorted:
+ items.sort()
+ items.reverse()
+
+ # Apply max_items filter
+ if len(items) and max_items:
+ items = items[:max_items]
+
+ # Apply max_days filter
+ if len(items) and max_days:
+ max_count = 0
+ max_time = items[0][0] - max_days * 84600
+ for item in items:
+ if item[0] > max_time:
+ max_count += 1
+ else:
+ items = items[:max_count]
+ break
+
+ return [ i[-1] for i in items ]
+
+class Channel(cache.CachedInfo):
+ """A list of news items.
+
+ This class represents a list of news items taken from the feed of
+ a website or other source.
+
+ Properties:
+ url URL of the feed.
+ url_etag E-Tag of the feed URL.
+ url_modified Last modified time of the feed URL.
+ url_status Last HTTP status of the feed URL.
+ hidden Channel should be hidden (True if exists).
+ name Name of the feed owner, or feed title.
+ next_order Next order number to be assigned to NewsItem
+
+ updated Correct UTC-Normalised update time of the feed.
+ last_updated Correct UTC-Normalised time the feed was last updated.
+
+ id An identifier the feed claims is unique (*).
+ title One-line title (*).
+ link Link to the original format feed (*).
+ tagline Short description of the feed (*).
+ info Longer description of the feed (*).
+
+ modified Date the feed claims to have been modified (*).
+
+ author Name of the author (*).
+ publisher Name of the publisher (*).
+ generator Name of the feed generator (*).
+ category Category name (*).
+ copyright Copyright information for humans to read (*).
+ license Link to the licence for the content (*).
+ docs Link to the specification of the feed format (*).
+ language Primary language (*).
+ errorreportsto E-Mail address to send error reports to (*).
+
+ image_url URL of an associated image (*).
+ image_link Link to go with the associated image (*).
+ image_title Alternative text of the associated image (*).
+ image_width Width of the associated image (*).
+ image_height Height of the associated image (*).
+
+ filter A regular expression that articles must match.
+ exclude A regular expression that articles must not match.
+
+ Properties marked (*) will only be present if the original feed
+ contained them. Note that the optional 'modified' date field is simply
+ a claim made by the item and parsed from the information given, 'updated'
+ (and 'last_updated') are far more reliable sources of information.
+
+ Some feeds may define additional properties to those above.
+ """
+ IGNORE_KEYS = ("links", "contributors", "textinput", "cloud", "categories",
+ "url", "href", "url_etag", "url_modified", "tags", "itunes_explicit")
+
+ def __init__(self, planet, url):
+ if not os.path.isdir(planet.cache_directory):
+ os.makedirs(planet.cache_directory)
+ cache_filename = cache.filename(planet.cache_directory, url)
+ cache_file = dbhash.open(cache_filename, "c", 0666)
+
+ cache.CachedInfo.__init__(self, cache_file, url, root=1)
+
+ self._items = {}
+ self._planet = planet
+ self._expired = []
+ self.url = url
+ # retain the original URL for error reporting
+ self.configured_url = url
+ self.url_etag = None
+ self.url_status = None
+ self.url_modified = None
+ self.name = None
+ self.updated = None
+ self.last_updated = None
+ self.filter = None
+ self.exclude = None
+ self.next_order = "0"
+ self.cache_read()
+ self.cache_read_entries()
+
+ if planet.config.has_section(url):
+ for option in planet.config.options(url):
+ value = planet.config.get(url, option)
+ self.set_as_string(option, value, cached=0)
+
+ def has_item(self, id_):
+ """Check whether the item exists in the channel."""
+ return self._items.has_key(id_)
+
+ def get_item(self, id_):
+ """Return the item from the channel."""
+ return self._items[id_]
+
+ # Special methods
+ __contains__ = has_item
+
+ def items(self, hidden=0, sorted=0):
+ """Return the item list."""
+ items = []
+ for item in self._items.values():
+ if hidden or not item.has_key("hidden"):
+ items.append((time.mktime(item.date), item.order, item))
+
+ if sorted:
+ items.sort()
+ items.reverse()
+
+ return [ i[-1] for i in items ]
+
+ def __iter__(self):
+ """Iterate the sorted item list."""
+ return iter(self.items(sorted=1))
+
+ def cache_read_entries(self):
+ """Read entry information from the cache."""
+ keys = self._cache.keys()
+ for key in keys:
+ if key.find(" ") != -1: continue
+ if self.has_key(key): continue
+
+ item = NewsItem(self, key)
+ self._items[key] = item
+
+ def cache_basename(self):
+ return cache.filename('',self._id)
+
+ def cache_write(self, sync=1):
+ """Write channel and item information to the cache."""
+ for item in self._items.values():
+ item.cache_write(sync=0)
+ for item in self._expired:
+ item.cache_clear(sync=0)
+ cache.CachedInfo.cache_write(self, sync)
+
+ self._expired = []
+
+ def feed_information(self):
+ """
+ Returns a description string for the feed embedded in this channel.
+
+ This will usually simply be the feed url embedded in <>, but in the
+ case where the current self.url has changed from the original
+ self.configured_url the string will contain both pieces of information.
+ This is so that the URL in question is easier to find in logging
+ output: getting an error about a URL that doesn't appear in your config
+ file is annoying.
+ """
+ if self.url == self.configured_url:
+ return "<%s>" % self.url
+ else:
+ return "<%s> (formerly <%s>)" % (self.url, self.configured_url)
+
+ def update(self):
+ """Download the feed to refresh the information.
+
+ This does the actual work of pulling down the feed and if it changes
+ updates the cached information about the feed and entries within it.
+ """
+ info = feedparser.parse(self.url,
+ etag=self.url_etag, modified=self.url_modified,
+ agent=self._planet.user_agent)
+ if info.has_key("status"):
+ self.url_status = str(info.status)
+ elif info.has_key("entries") and len(info.entries)>0:
+ self.url_status = str(200)
+ elif info.bozo and info.bozo_exception.__class__.__name__=='Timeout':
+ self.url_status = str(408)
+ else:
+ self.url_status = str(500)
+
+ if self.url_status == '301' and \
+ (info.has_key("entries") and len(info.entries)>0):
+ log.warning("Feed has moved from <%s> to <%s>", self.url, info.url)
+ try:
+ os.link(cache.filename(self._planet.cache_directory, self.url),
+ cache.filename(self._planet.cache_directory, info.url))
+ except:
+ pass
+ self.url = info.url
+ elif self.url_status == '304':
+ log.info("Feed %s unchanged", self.feed_information())
+ return
+ elif self.url_status == '410':
+ log.info("Feed %s gone", self.feed_information())
+ self.cache_write()
+ return
+ elif self.url_status == '408':
+ log.warning("Feed %s timed out", self.feed_information())
+ return
+ elif int(self.url_status) >= 400:
+ log.error("Error %s while updating feed %s",
+ self.url_status, self.feed_information())
+ return
+ else:
+ log.info("Updating feed %s", self.feed_information())
+
+ self.url_etag = info.has_key("etag") and info.etag or None
+ self.url_modified = info.has_key("modified") and info.modified or None
+ if self.url_etag is not None:
+ log.debug("E-Tag: %s", self.url_etag)
+ if self.url_modified is not None:
+ log.debug("Last Modified: %s",
+ time.strftime(TIMEFMT_ISO, self.url_modified))
+
+ self.update_info(info.feed)
+ self.update_entries(info.entries)
+ self.cache_write()
+
+ def update_info(self, feed):
+ """Update information from the feed.
+
+ This reads the feed information supplied by feedparser and updates
+ the cached information about the feed. These are the various
+ potentially interesting properties that you might care about.
+ """
+ for key in feed.keys():
+ if key in self.IGNORE_KEYS or key + "_parsed" in self.IGNORE_KEYS:
+ # Ignored fields
+ pass
+ elif feed.has_key(key + "_parsed"):
+ # Ignore unparsed date fields
+ pass
+ elif key.endswith("_detail"):
+ # retain name and email sub-fields
+ if feed[key].has_key('name') and feed[key].name:
+ self.set_as_string(key.replace("_detail","_name"), \
+ feed[key].name)
+ if feed[key].has_key('email') and feed[key].email:
+ self.set_as_string(key.replace("_detail","_email"), \
+ feed[key].email)
+ elif key == "items":
+ # Ignore items field
+ pass
+ elif key.endswith("_parsed"):
+ # Date fields
+ if feed[key] is not None:
+ self.set_as_date(key[:-len("_parsed")], feed[key])
+ elif key == "image":
+ # Image field: save all the information
+ if feed[key].has_key("url"):
+ self.set_as_string(key + "_url", feed[key].url)
+ if feed[key].has_key("link"):
+ self.set_as_string(key + "_link", feed[key].link)
+ if feed[key].has_key("title"):
+ self.set_as_string(key + "_title", feed[key].title)
+ if feed[key].has_key("width"):
+ self.set_as_string(key + "_width", str(feed[key].width))
+ if feed[key].has_key("height"):
+ self.set_as_string(key + "_height", str(feed[key].height))
+ elif isinstance(feed[key], (str, unicode)):
+ # String fields
+ try:
+ detail = key + '_detail'
+ if feed.has_key(detail) and feed[detail].has_key('type'):
+ if feed[detail].type == 'text/html':
+ feed[key] = sanitize.HTML(feed[key])
+ elif feed[detail].type == 'text/plain':
+ feed[key] = escape(feed[key])
+ self.set_as_string(key, feed[key])
+ except KeyboardInterrupt:
+ raise
+ except:
+ log.exception("Ignored '%s' of <%s>, unknown format",
+ key, self.url)
+
+ def update_entries(self, entries):
+ """Update entries from the feed.
+
+ This reads the entries supplied by feedparser and updates the
+ cached information about them. It's at this point we update
+ the 'updated' timestamp and keep the old one in 'last_updated',
+ these provide boundaries for acceptable entry times.
+
+ If this is the first time a feed has been updated then most of the
+ items will be marked as hidden, according to Planet.new_feed_items.
+
+ If the feed does not contain items which, according to the sort order,
+ should be there; those items are assumed to have been expired from
+ the feed or replaced and are removed from the cache.
+ """
+ if not len(entries):
+ return
+
+ self.last_updated = self.updated
+ self.updated = time.gmtime()
+
+ new_items = []
+ feed_items = []
+ for entry in entries:
+ # Try really hard to find some kind of unique identifier
+ if entry.has_key("id"):
+ entry_id = cache.utf8(entry.id)
+ elif entry.has_key("link"):
+ entry_id = cache.utf8(entry.link)
+ elif entry.has_key("title"):
+ entry_id = (self.url + "/"
+ + md5.new(cache.utf8(entry.title)).hexdigest())
+ elif entry.has_key("summary"):
+ entry_id = (self.url + "/"
+ + md5.new(cache.utf8(entry.summary)).hexdigest())
+ else:
+ log.error("Unable to find or generate id, entry ignored")
+ continue
+
+ # Create the item if necessary and update
+ if self.has_item(entry_id):
+ item = self._items[entry_id]
+ else:
+ item = NewsItem(self, entry_id)
+ self._items[entry_id] = item
+ new_items.append(item)
+ item.update(entry)
+ feed_items.append(entry_id)
+
+ # Hide excess items the first time through
+ if self.last_updated is None and self._planet.new_feed_items \
+ and len(feed_items) > self._planet.new_feed_items:
+ item.hidden = "yes"
+ log.debug("Marked <%s> as hidden (new feed)", entry_id)
+
+ # Assign order numbers in reverse
+ new_items.reverse()
+ for item in new_items:
+ item.order = self.next_order = str(int(self.next_order) + 1)
+
+ # Check for expired or replaced items
+ feed_count = len(feed_items)
+ log.debug("Items in Feed: %d", feed_count)
+ for item in self.items(sorted=1):
+ if feed_count < 1:
+ break
+ elif item.id in feed_items:
+ feed_count -= 1
+ elif item._channel.url_status != '226':
+ del(self._items[item.id])
+ self._expired.append(item)
+ log.debug("Removed expired or replaced item <%s>", item.id)
+
+ def get_name(self, key):
+ """Return the key containing the name."""
+ for key in ("name", "title"):
+ if self.has_key(key) and self.key_type(key) != self.NULL:
+ return self.get_as_string(key)
+
+ return ""
+
+class NewsItem(cache.CachedInfo):
+ """An item of news.
+
+ This class represents a single item of news on a channel. They're
+ created by members of the Channel class and accessible through it.
+
+ Properties:
+ id Channel-unique identifier for this item.
+ id_hash Relatively short, printable cryptographic hash of id
+ date Corrected UTC-Normalised update time, for sorting.
+ order Order in which items on the same date can be sorted.
+ hidden Item should be hidden (True if exists).
+
+ title One-line title (*).
+ link Link to the original format text (*).
+ summary Short first-page summary (*).
+ content Full HTML content.
+
+ modified Date the item claims to have been modified (*).
+ issued Date the item claims to have been issued (*).
+ created Date the item claims to have been created (*).
+ expired Date the item claims to expire (*).
+
+ author Name of the author (*).
+ publisher Name of the publisher (*).
+ category Category name (*).
+ comments Link to a page to enter comments (*).
+ license Link to the licence for the content (*).
+ source_name Name of the original source of this item (*).
+ source_link Link to the original source of this item (*).
+
+ Properties marked (*) will only be present if the original feed
+ contained them. Note that the various optional date fields are
+ simply claims made by the item and parsed from the information
+ given, 'date' is a far more reliable source of information.
+
+ Some feeds may define additional properties to those above.
+ """
+ IGNORE_KEYS = ("categories", "contributors", "enclosures", "links",
+ "guidislink", "date", "tags")
+
+ def __init__(self, channel, id_):
+ cache.CachedInfo.__init__(self, channel._cache, id_)
+
+ self._channel = channel
+ self.id = id_
+ self.id_hash = md5.new(id_).hexdigest()
+ self.date = None
+ self.order = None
+ self.content = None
+ self.cache_read()
+
+ def update(self, entry):
+ """Update the item from the feedparser entry given."""
+ for key in entry.keys():
+ if key in self.IGNORE_KEYS or key + "_parsed" in self.IGNORE_KEYS:
+ # Ignored fields
+ pass
+ elif entry.has_key(key + "_parsed"):
+ # Ignore unparsed date fields
+ pass
+ elif key.endswith("_detail"):
+ # retain name, email, and language sub-fields
+ if entry[key].has_key('name') and entry[key].name:
+ self.set_as_string(key.replace("_detail","_name"), \
+ entry[key].name)
+ if entry[key].has_key('email') and entry[key].email:
+ self.set_as_string(key.replace("_detail","_email"), \
+ entry[key].email)
+ if entry[key].has_key('language') and entry[key].language and \
+ (not self._channel.has_key('language') or \
+ entry[key].language != self._channel.language):
+ self.set_as_string(key.replace("_detail","_language"), \
+ entry[key].language)
+ elif key.endswith("_parsed"):
+ # Date fields
+ if entry[key] is not None:
+ self.set_as_date(key[:-len("_parsed")], entry[key])
+ elif key == "source":
+ # Source field: save both url and value
+ if entry[key].has_key("value"):
+ self.set_as_string(key + "_name", entry[key].value)
+ if entry[key].has_key("url"):
+ self.set_as_string(key + "_link", entry[key].url)
+ elif key == "content":
+ # Content field: concatenate the values
+ value = ""
+ for item in entry[key]:
+ if item.type == 'text/html':
+ item.value = sanitize.HTML(item.value)
+ elif item.type == 'text/plain':
+ item.value = escape(item.value)
+ if item.has_key('language') and item.language and \
+ (not self._channel.has_key('language') or
+ item.language != self._channel.language) :
+ self.set_as_string(key + "_language", item.language)
+ value += cache.utf8(item.value)
+ self.set_as_string(key, value)
+ elif isinstance(entry[key], (str, unicode)):
+ # String fields
+ try:
+ detail = key + '_detail'
+ if entry.has_key(detail):
+ if entry[detail].has_key('type'):
+ if entry[detail].type == 'text/html':
+ entry[key] = sanitize.HTML(entry[key])
+ elif entry[detail].type == 'text/plain':
+ entry[key] = escape(entry[key])
+ self.set_as_string(key, entry[key])
+ except KeyboardInterrupt:
+ raise
+ except:
+ log.exception("Ignored '%s' of <%s>, unknown format",
+ key, self.id)
+
+ # Generate the date field if we need to
+ self.get_date("date")
+
+ def get_date(self, key):
+ """Get (or update) the date key.
+
+ We check whether the date the entry claims to have been changed is
+ since we last updated this feed and when we pulled the feed off the
+ site.
+
+ If it is then it's probably not bogus, and we'll sort accordingly.
+
+ If it isn't then we bound it appropriately, this ensures that
+ entries appear in posting sequence but don't overlap entries
+ added in previous updates and don't creep into the next one.
+ """
+
+ for other_key in ("updated", "modified", "published", "issued", "created"):
+ if self.has_key(other_key):
+ date = self.get_as_date(other_key)
+ break
+ else:
+ date = None
+
+ if date is not None:
+ if date > self._channel.updated:
+ date = self._channel.updated
+# elif date < self._channel.last_updated:
+# date = self._channel.updated
+ elif self.has_key(key) and self.key_type(key) != self.NULL:
+ return self.get_as_date(key)
+ else:
+ date = self._channel.updated
+
+ self.set_as_date(key, date)
+ return date
+
+ def get_content(self, key):
+ """Return the key containing the content."""
+ for key in ("content", "tagline", "summary"):
+ if self.has_key(key) and self.key_type(key) != self.NULL:
+ return self.get_as_string(key)
+
+ return ""
diff --git a/code/planet/atomstyler.py b/code/planet/atomstyler.py
new file mode 100644
index 0000000..9220702
--- /dev/null
+++ b/code/planet/atomstyler.py
@@ -0,0 +1,124 @@
+from xml.dom import minidom, Node
+from urlparse import urlparse, urlunparse
+from xml.parsers.expat import ExpatError
+from htmlentitydefs import name2codepoint
+import re
+
+# select and apply an xml:base for this entry
+class relativize:
+ def __init__(self, parent):
+ self.score = {}
+ self.links = []
+ self.collect_and_tally(parent)
+ self.base = self.select_optimal_base()
+ if self.base:
+ if not parent.hasAttribute('xml:base'):
+ self.rebase(parent)
+ parent.setAttribute('xml:base', self.base)
+
+ # collect and tally cite, href and src attributes
+ def collect_and_tally(self,parent):
+ uri = None
+ if parent.hasAttribute('cite'): uri=parent.getAttribute('cite')
+ if parent.hasAttribute('href'): uri=parent.getAttribute('href')
+ if parent.hasAttribute('src'): uri=parent.getAttribute('src')
+
+ if uri:
+ parts=urlparse(uri)
+ if parts[0].lower() == 'http':
+ parts = (parts[1]+parts[2]).split('/')
+ base = None
+ for i in range(1,len(parts)):
+ base = tuple(parts[0:i])
+ self.score[base] = self.score.get(base,0) + len(base)
+ if base and base not in self.links: self.links.append(base)
+
+ for node in parent.childNodes:
+ if node.nodeType == Node.ELEMENT_NODE:
+ self.collect_and_tally(node)
+
+ # select the xml:base with the highest score
+ def select_optimal_base(self):
+ if not self.score: return None
+ for link in self.links:
+ self.score[link] = 0
+ winner = max(self.score.values())
+ if not winner: return None
+ for key in self.score.keys():
+ if self.score[key] == winner:
+ if winner == len(key): return None
+ return urlunparse(('http', key[0], '/'.join(key[1:]), '', '', '')) + '/'
+
+ # rewrite cite, href and src attributes using this base
+ def rebase(self,parent):
+ uri = None
+ if parent.hasAttribute('cite'): uri=parent.getAttribute('cite')
+ if parent.hasAttribute('href'): uri=parent.getAttribute('href')
+ if parent.hasAttribute('src'): uri=parent.getAttribute('src')
+ if uri and uri.startswith(self.base):
+ uri = uri[len(self.base):] or '.'
+ if parent.hasAttribute('href'): uri=parent.setAttribute('href', uri)
+ if parent.hasAttribute('src'): uri=parent.setAttribute('src', uri)
+
+ for node in parent.childNodes:
+ if node.nodeType == Node.ELEMENT_NODE:
+ self.rebase(node)
+
+# convert type="html" to type="plain" or type="xhtml" as appropriate
+def retype(parent):
+ for node in parent.childNodes:
+ if node.nodeType == Node.ELEMENT_NODE:
+
+ if node.hasAttribute('type') and node.getAttribute('type') == 'html':
+ if len(node.childNodes)==0:
+ node.removeAttribute('type')
+ elif len(node.childNodes)==1:
+
+ # replace html entity defs with utf-8
+ chunks=re.split('&(\w+);', node.childNodes[0].nodeValue)
+ for i in range(1,len(chunks),2):
+ if chunks[i] in ['amp', 'lt', 'gt', 'apos', 'quot']:
+ chunks[i] ='&' + chunks[i] +';'
+ elif chunks[i] in name2codepoint:
+ chunks[i]=unichr(name2codepoint[chunks[i]])
+ else:
+ chunks[i]='&' + chunks[i] + ';'
+ text = u"".join(chunks)
+
+ try:
+ # see if the resulting text is a well-formed XML fragment
+ div = '
and
+ # only if all the remaining content is nested underneath it.
+ # This means that the divs would be retained in the following:
+ #
foo
bar
+ if pieces and (pieces[0] == '
' or pieces[0].startswith('
':
+ depth = 0
+ for piece in pieces[:-1]:
+ if piece.startswith(''):
+ depth -= 1
+ if depth == 0: break
+ elif piece.startswith('<') and not piece.endswith('/>'):
+ depth += 1
+ else:
+ pieces = pieces[1:-1]
+
+ output = ''.join(pieces)
+ if stripWhitespace:
+ output = output.strip()
+ if not expectingText: return output
+
+ # decode base64 content
+ if base64 and self.contentparams.get('base64', 0):
+ try:
+ output = base64.decodestring(output)
+ except binascii.Error:
+ pass
+ except binascii.Incomplete:
+ pass
+
+ # resolve relative URIs
+ if (element in self.can_be_relative_uri) and output:
+ output = self.resolveURI(output)
+
+ # decode entities within embedded markup
+ if not self.contentparams.get('base64', 0):
+ output = self.decodeEntities(element, output)
+
+ # remove temporary cruft from contentparams
+ try:
+ del self.contentparams['mode']
+ except KeyError:
+ pass
+ try:
+ del self.contentparams['base64']
+ except KeyError:
+ pass
+
+ # resolve relative URIs within embedded markup
+ if self.mapContentType(self.contentparams.get('type', 'text/html')) in self.html_types:
+ if element in self.can_contain_relative_uris:
+ output = _resolveRelativeURIs(output, self.baseuri, self.encoding)
+
+ # sanitize embedded markup
+ if self.mapContentType(self.contentparams.get('type', 'text/html')) in self.html_types:
+ if element in self.can_contain_dangerous_markup:
+ output = _sanitizeHTML(output, self.encoding)
+
+ if self.encoding and type(output) != type(u''):
+ try:
+ output = unicode(output, self.encoding)
+ except:
+ pass
+
+ # address common error where people take data that is already
+ # utf-8, presume that it is iso-8859-1, and re-encode it.
+ if self.encoding=='utf-8' and type(output) == type(u''):
+ try:
+ output = unicode(output.encode('iso-8859-1'), 'utf-8')
+ except:
+ pass
+
+ # map win-1252 extensions to the proper code points
+ if type(output) == type(u''):
+ output = u''.join([c in cp1252 and cp1252[c] or c for c in output])
+
+ # categories/tags/keywords/whatever are handled in _end_category
+ if element == 'category':
+ return output
+
+ # store output in appropriate place(s)
+ if self.inentry and not self.insource:
+ if element == 'content':
+ self.entries[-1].setdefault(element, [])
+ contentparams = copy.deepcopy(self.contentparams)
+ contentparams['value'] = output
+ self.entries[-1][element].append(contentparams)
+ elif element == 'link':
+ self.entries[-1][element] = output
+ if output:
+ self.entries[-1]['links'][-1]['href'] = output
+ else:
+ if element == 'description':
+ element = 'summary'
+ self.entries[-1][element] = output
+ if self.incontent:
+ contentparams = copy.deepcopy(self.contentparams)
+ contentparams['value'] = output
+ self.entries[-1][element + '_detail'] = contentparams
+ elif (self.infeed or self.insource) and (not self.intextinput) and (not self.inimage):
+ context = self._getContext()
+ if element == 'description':
+ element = 'subtitle'
+ context[element] = output
+ if element == 'link':
+ context['links'][-1]['href'] = output
+ elif self.incontent:
+ contentparams = copy.deepcopy(self.contentparams)
+ contentparams['value'] = output
+ context[element + '_detail'] = contentparams
+ return output
+
+ def pushContent(self, tag, attrsD, defaultContentType, expectingText):
+ self.incontent += 1
+ self.contentparams = FeedParserDict({
+ 'type': self.mapContentType(attrsD.get('type', defaultContentType)),
+ 'language': self.lang,
+ 'base': self.baseuri})
+ self.contentparams['base64'] = self._isBase64(attrsD, self.contentparams)
+ self.push(tag, expectingText)
+
+ def popContent(self, tag):
+ value = self.pop(tag)
+ self.incontent -= 1
+ self.contentparams.clear()
+ return value
+
+ def _mapToStandardPrefix(self, name):
+ colonpos = name.find(':')
+ if colonpos <> -1:
+ prefix = name[:colonpos]
+ suffix = name[colonpos+1:]
+ prefix = self.namespacemap.get(prefix, prefix)
+ name = prefix + ':' + suffix
+ return name
+
+ def _getAttribute(self, attrsD, name):
+ return attrsD.get(self._mapToStandardPrefix(name))
+
+ def _isBase64(self, attrsD, contentparams):
+ if attrsD.get('mode', '') == 'base64':
+ return 1
+ if self.contentparams['type'].startswith('text/'):
+ return 0
+ if self.contentparams['type'].endswith('+xml'):
+ return 0
+ if self.contentparams['type'].endswith('/xml'):
+ return 0
+ return 1
+
+ def _itsAnHrefDamnIt(self, attrsD):
+ href = attrsD.get('url', attrsD.get('uri', attrsD.get('href', None)))
+ if href:
+ try:
+ del attrsD['url']
+ except KeyError:
+ pass
+ try:
+ del attrsD['uri']
+ except KeyError:
+ pass
+ attrsD['href'] = href
+ return attrsD
+
+ def _save(self, key, value):
+ context = self._getContext()
+ context.setdefault(key, value)
+
+ def _start_rss(self, attrsD):
+ versionmap = {'0.91': 'rss091u',
+ '0.92': 'rss092',
+ '0.93': 'rss093',
+ '0.94': 'rss094'}
+ if not self.version:
+ attr_version = attrsD.get('version', '')
+ version = versionmap.get(attr_version)
+ if version:
+ self.version = version
+ elif attr_version.startswith('2.'):
+ self.version = 'rss20'
+ else:
+ self.version = 'rss'
+
+ def _start_dlhottitles(self, attrsD):
+ self.version = 'hotrss'
+
+ def _start_channel(self, attrsD):
+ self.infeed = 1
+ self._cdf_common(attrsD)
+ _start_feedinfo = _start_channel
+
+ def _cdf_common(self, attrsD):
+ if attrsD.has_key('lastmod'):
+ self._start_modified({})
+ self.elementstack[-1][-1] = attrsD['lastmod']
+ self._end_modified()
+ if attrsD.has_key('href'):
+ self._start_link({})
+ self.elementstack[-1][-1] = attrsD['href']
+ self._end_link()
+
+ def _start_feed(self, attrsD):
+ self.infeed = 1
+ versionmap = {'0.1': 'atom01',
+ '0.2': 'atom02',
+ '0.3': 'atom03'}
+ if not self.version:
+ attr_version = attrsD.get('version')
+ version = versionmap.get(attr_version)
+ if version:
+ self.version = version
+ else:
+ self.version = 'atom'
+
+ def _end_channel(self):
+ self.infeed = 0
+ _end_feed = _end_channel
+
+ def _start_image(self, attrsD):
+ self.inimage = 1
+ self.push('image', 0)
+ context = self._getContext()
+ context.setdefault('image', FeedParserDict())
+
+ def _end_image(self):
+ self.pop('image')
+ self.inimage = 0
+
+ def _start_textinput(self, attrsD):
+ self.intextinput = 1
+ self.push('textinput', 0)
+ context = self._getContext()
+ context.setdefault('textinput', FeedParserDict())
+ _start_textInput = _start_textinput
+
+ def _end_textinput(self):
+ self.pop('textinput')
+ self.intextinput = 0
+ _end_textInput = _end_textinput
+
+ def _start_author(self, attrsD):
+ self.inauthor = 1
+ self.push('author', 1)
+ _start_managingeditor = _start_author
+ _start_dc_author = _start_author
+ _start_dc_creator = _start_author
+ _start_itunes_author = _start_author
+
+ def _end_author(self):
+ self.pop('author')
+ self.inauthor = 0
+ self._sync_author_detail()
+ _end_managingeditor = _end_author
+ _end_dc_author = _end_author
+ _end_dc_creator = _end_author
+ _end_itunes_author = _end_author
+
+ def _start_itunes_owner(self, attrsD):
+ self.inpublisher = 1
+ self.push('publisher', 0)
+
+ def _end_itunes_owner(self):
+ self.pop('publisher')
+ self.inpublisher = 0
+ self._sync_author_detail('publisher')
+
+ def _start_contributor(self, attrsD):
+ self.incontributor = 1
+ context = self._getContext()
+ context.setdefault('contributors', [])
+ context['contributors'].append(FeedParserDict())
+ self.push('contributor', 0)
+
+ def _end_contributor(self):
+ self.pop('contributor')
+ self.incontributor = 0
+
+ def _start_dc_contributor(self, attrsD):
+ self.incontributor = 1
+ context = self._getContext()
+ context.setdefault('contributors', [])
+ context['contributors'].append(FeedParserDict())
+ self.push('name', 0)
+
+ def _end_dc_contributor(self):
+ self._end_name()
+ self.incontributor = 0
+
+ def _start_name(self, attrsD):
+ self.push('name', 0)
+ _start_itunes_name = _start_name
+
+ def _end_name(self):
+ value = self.pop('name')
+ if self.inpublisher:
+ self._save_author('name', value, 'publisher')
+ elif self.inauthor:
+ self._save_author('name', value)
+ elif self.incontributor:
+ self._save_contributor('name', value)
+ elif self.intextinput:
+ context = self._getContext()
+ context['textinput']['name'] = value
+ _end_itunes_name = _end_name
+
+ def _start_width(self, attrsD):
+ self.push('width', 0)
+
+ def _end_width(self):
+ value = self.pop('width')
+ try:
+ value = int(value)
+ except:
+ value = 0
+ if self.inimage:
+ context = self._getContext()
+ context['image']['width'] = value
+
+ def _start_height(self, attrsD):
+ self.push('height', 0)
+
+ def _end_height(self):
+ value = self.pop('height')
+ try:
+ value = int(value)
+ except:
+ value = 0
+ if self.inimage:
+ context = self._getContext()
+ context['image']['height'] = value
+
+ def _start_url(self, attrsD):
+ self.push('href', 1)
+ _start_homepage = _start_url
+ _start_uri = _start_url
+
+ def _end_url(self):
+ value = self.pop('href')
+ if self.inauthor:
+ self._save_author('href', value)
+ elif self.incontributor:
+ self._save_contributor('href', value)
+ elif self.inimage:
+ context = self._getContext()
+ context['image']['href'] = value
+ elif self.intextinput:
+ context = self._getContext()
+ context['textinput']['link'] = value
+ _end_homepage = _end_url
+ _end_uri = _end_url
+
+ def _start_email(self, attrsD):
+ self.push('email', 0)
+ _start_itunes_email = _start_email
+
+ def _end_email(self):
+ value = self.pop('email')
+ if self.inpublisher:
+ self._save_author('email', value, 'publisher')
+ elif self.inauthor:
+ self._save_author('email', value)
+ elif self.incontributor:
+ self._save_contributor('email', value)
+ _end_itunes_email = _end_email
+
+ def _getContext(self):
+ if self.insource:
+ context = self.sourcedata
+ elif self.inentry:
+ context = self.entries[-1]
+ else:
+ context = self.feeddata
+ return context
+
+ def _save_author(self, key, value, prefix='author'):
+ context = self._getContext()
+ context.setdefault(prefix + '_detail', FeedParserDict())
+ context[prefix + '_detail'][key] = value
+ self._sync_author_detail()
+
+ def _save_contributor(self, key, value):
+ context = self._getContext()
+ context.setdefault('contributors', [FeedParserDict()])
+ context['contributors'][-1][key] = value
+
+ def _sync_author_detail(self, key='author'):
+ context = self._getContext()
+ detail = context.get('%s_detail' % key)
+ if detail:
+ name = detail.get('name')
+ email = detail.get('email')
+ if name and email:
+ context[key] = '%s (%s)' % (name, email)
+ elif name:
+ context[key] = name
+ elif email:
+ context[key] = email
+ else:
+ author = context.get(key)
+ if not author: return
+ emailmatch = re.search(r'''(([a-zA-Z0-9\_\-\.\+]+)@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.)|(([a-zA-Z0-9\-]+\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\]?))''', author)
+ if not emailmatch: return
+ email = emailmatch.group(0)
+ # probably a better way to do the following, but it passes all the tests
+ author = author.replace(email, '')
+ author = author.replace('()', '')
+ author = author.strip()
+ if author and (author[0] == '('):
+ author = author[1:]
+ if author and (author[-1] == ')'):
+ author = author[:-1]
+ author = author.strip()
+ context.setdefault('%s_detail' % key, FeedParserDict())
+ context['%s_detail' % key]['name'] = author
+ context['%s_detail' % key]['email'] = email
+
+ def _start_subtitle(self, attrsD):
+ self.pushContent('subtitle', attrsD, 'text/plain', 1)
+ _start_tagline = _start_subtitle
+ _start_itunes_subtitle = _start_subtitle
+
+ def _end_subtitle(self):
+ self.popContent('subtitle')
+ _end_tagline = _end_subtitle
+ _end_itunes_subtitle = _end_subtitle
+
+ def _start_rights(self, attrsD):
+ self.pushContent('rights', attrsD, 'text/plain', 1)
+ _start_dc_rights = _start_rights
+ _start_copyright = _start_rights
+
+ def _end_rights(self):
+ self.popContent('rights')
+ _end_dc_rights = _end_rights
+ _end_copyright = _end_rights
+
+ def _start_item(self, attrsD):
+ self.entries.append(FeedParserDict())
+ self.push('item', 0)
+ self.inentry = 1
+ self.guidislink = 0
+ id = self._getAttribute(attrsD, 'rdf:about')
+ if id:
+ context = self._getContext()
+ context['id'] = id
+ self._cdf_common(attrsD)
+ _start_entry = _start_item
+ _start_product = _start_item
+
+ def _end_item(self):
+ self.pop('item')
+ self.inentry = 0
+ _end_entry = _end_item
+
+ def _start_dc_language(self, attrsD):
+ self.push('language', 1)
+ _start_language = _start_dc_language
+
+ def _end_dc_language(self):
+ self.lang = self.pop('language')
+ _end_language = _end_dc_language
+
+ def _start_dc_publisher(self, attrsD):
+ self.push('publisher', 1)
+ _start_webmaster = _start_dc_publisher
+
+ def _end_dc_publisher(self):
+ self.pop('publisher')
+ self._sync_author_detail('publisher')
+ _end_webmaster = _end_dc_publisher
+
+ def _start_published(self, attrsD):
+ self.push('published', 1)
+ _start_dcterms_issued = _start_published
+ _start_issued = _start_published
+
+ def _end_published(self):
+ value = self.pop('published')
+ self._save('published_parsed', _parse_date(value))
+ _end_dcterms_issued = _end_published
+ _end_issued = _end_published
+
+ def _start_updated(self, attrsD):
+ self.push('updated', 1)
+ _start_modified = _start_updated
+ _start_dcterms_modified = _start_updated
+ _start_pubdate = _start_updated
+ _start_dc_date = _start_updated
+
+ def _end_updated(self):
+ value = self.pop('updated')
+ parsed_value = _parse_date(value)
+ self._save('updated_parsed', parsed_value)
+ _end_modified = _end_updated
+ _end_dcterms_modified = _end_updated
+ _end_pubdate = _end_updated
+ _end_dc_date = _end_updated
+
+ def _start_created(self, attrsD):
+ self.push('created', 1)
+ _start_dcterms_created = _start_created
+
+ def _end_created(self):
+ value = self.pop('created')
+ self._save('created_parsed', _parse_date(value))
+ _end_dcterms_created = _end_created
+
+ def _start_expirationdate(self, attrsD):
+ self.push('expired', 1)
+
+ def _end_expirationdate(self):
+ self._save('expired_parsed', _parse_date(self.pop('expired')))
+
+ def _start_cc_license(self, attrsD):
+ self.push('license', 1)
+ value = self._getAttribute(attrsD, 'rdf:resource')
+ if value:
+ self.elementstack[-1][2].append(value)
+ self.pop('license')
+
+ def _start_creativecommons_license(self, attrsD):
+ self.push('license', 1)
+
+ def _end_creativecommons_license(self):
+ self.pop('license')
+
+ def _addTag(self, term, scheme, label):
+ context = self._getContext()
+ tags = context.setdefault('tags', [])
+ if (not term) and (not scheme) and (not label): return
+ value = FeedParserDict({'term': term, 'scheme': scheme, 'label': label})
+ if value not in tags:
+ tags.append(FeedParserDict({'term': term, 'scheme': scheme, 'label': label}))
+
+ def _start_category(self, attrsD):
+ if _debug: sys.stderr.write('entering _start_category with %s\n' % repr(attrsD))
+ term = attrsD.get('term')
+ scheme = attrsD.get('scheme', attrsD.get('domain'))
+ label = attrsD.get('label')
+ self._addTag(term, scheme, label)
+ self.push('category', 1)
+ _start_dc_subject = _start_category
+ _start_keywords = _start_category
+
+ def _end_itunes_keywords(self):
+ for term in self.pop('itunes_keywords').split():
+ self._addTag(term, 'http://www.itunes.com/', None)
+
+ def _start_itunes_category(self, attrsD):
+ self._addTag(attrsD.get('text'), 'http://www.itunes.com/', None)
+ self.push('category', 1)
+
+ def _end_category(self):
+ value = self.pop('category')
+ if not value: return
+ context = self._getContext()
+ tags = context['tags']
+ if value and len(tags) and not tags[-1]['term']:
+ tags[-1]['term'] = value
+ else:
+ self._addTag(value, None, None)
+ _end_dc_subject = _end_category
+ _end_keywords = _end_category
+ _end_itunes_category = _end_category
+
+ def _start_cloud(self, attrsD):
+ self._getContext()['cloud'] = FeedParserDict(attrsD)
+
+ def _start_link(self, attrsD):
+ attrsD.setdefault('rel', 'alternate')
+ attrsD.setdefault('type', 'text/html')
+ attrsD = self._itsAnHrefDamnIt(attrsD)
+ if attrsD.has_key('href'):
+ attrsD['href'] = self.resolveURI(attrsD['href'])
+ expectingText = self.infeed or self.inentry or self.insource
+ context = self._getContext()
+ context.setdefault('links', [])
+ context['links'].append(FeedParserDict(attrsD))
+ if attrsD['rel'] == 'enclosure':
+ self._start_enclosure(attrsD)
+ if attrsD.has_key('href'):
+ expectingText = 0
+ if (attrsD.get('rel') == 'alternate') and (self.mapContentType(attrsD.get('type')) in self.html_types):
+ context['link'] = attrsD['href']
+ else:
+ self.push('link', expectingText)
+ _start_producturl = _start_link
+
+ def _end_link(self):
+ value = self.pop('link')
+ context = self._getContext()
+ if self.intextinput:
+ context['textinput']['link'] = value
+ if self.inimage:
+ context['image']['link'] = value
+ _end_producturl = _end_link
+
+ def _start_guid(self, attrsD):
+ self.guidislink = (attrsD.get('ispermalink', 'true') == 'true')
+ self.push('id', 1)
+
+ def _end_guid(self):
+ value = self.pop('id')
+ self._save('guidislink', self.guidislink and not self._getContext().has_key('link'))
+ if self.guidislink:
+ # guid acts as link, but only if 'ispermalink' is not present or is 'true',
+ # and only if the item doesn't already have a link element
+ self._save('link', value)
+
+ def _start_title(self, attrsD):
+ self.pushContent('title', attrsD, 'text/plain', self.infeed or self.inentry or self.insource)
+ def _start_title_low_pri(self, attrsD):
+ if not self._getContext().has_key('title'):
+ self._start_title(attrsD)
+ _start_dc_title = _start_title_low_pri
+ _start_media_title = _start_title_low_pri
+
+ def _end_title(self):
+ value = self.popContent('title')
+ context = self._getContext()
+ if self.intextinput:
+ context['textinput']['title'] = value
+ elif self.inimage:
+ context['image']['title'] = value
+ def _end_title_low_pri(self):
+ if not self._getContext().has_key('title'):
+ self._end_title()
+ _end_dc_title = _end_title_low_pri
+ _end_media_title = _end_title_low_pri
+
+ def _start_description(self, attrsD):
+ context = self._getContext()
+ if context.has_key('summary'):
+ self._summaryKey = 'content'
+ self._start_content(attrsD)
+ else:
+ self.pushContent('description', attrsD, 'text/html', self.infeed or self.inentry or self.insource)
+
+ def _start_abstract(self, attrsD):
+ self.pushContent('description', attrsD, 'text/plain', self.infeed or self.inentry or self.insource)
+
+ def _end_description(self):
+ if self._summaryKey == 'content':
+ self._end_content()
+ else:
+ value = self.popContent('description')
+ context = self._getContext()
+ if self.intextinput:
+ context['textinput']['description'] = value
+ elif self.inimage:
+ context['image']['description'] = value
+ self._summaryKey = None
+ _end_abstract = _end_description
+
+ def _start_info(self, attrsD):
+ self.pushContent('info', attrsD, 'text/plain', 1)
+ _start_feedburner_browserfriendly = _start_info
+
+ def _end_info(self):
+ self.popContent('info')
+ _end_feedburner_browserfriendly = _end_info
+
+ def _start_generator(self, attrsD):
+ if attrsD:
+ attrsD = self._itsAnHrefDamnIt(attrsD)
+ if attrsD.has_key('href'):
+ attrsD['href'] = self.resolveURI(attrsD['href'])
+ self._getContext()['generator_detail'] = FeedParserDict(attrsD)
+ self.push('generator', 1)
+
+ def _end_generator(self):
+ value = self.pop('generator')
+ context = self._getContext()
+ if context.has_key('generator_detail'):
+ context['generator_detail']['name'] = value
+
+ def _start_admin_generatoragent(self, attrsD):
+ self.push('generator', 1)
+ value = self._getAttribute(attrsD, 'rdf:resource')
+ if value:
+ self.elementstack[-1][2].append(value)
+ self.pop('generator')
+ self._getContext()['generator_detail'] = FeedParserDict({'href': value})
+
+ def _start_admin_errorreportsto(self, attrsD):
+ self.push('errorreportsto', 1)
+ value = self._getAttribute(attrsD, 'rdf:resource')
+ if value:
+ self.elementstack[-1][2].append(value)
+ self.pop('errorreportsto')
+
+ def _start_summary(self, attrsD):
+ context = self._getContext()
+ if context.has_key('summary'):
+ self._summaryKey = 'content'
+ self._start_content(attrsD)
+ else:
+ self._summaryKey = 'summary'
+ self.pushContent(self._summaryKey, attrsD, 'text/plain', 1)
+ _start_itunes_summary = _start_summary
+
+ def _end_summary(self):
+ if self._summaryKey == 'content':
+ self._end_content()
+ else:
+ self.popContent(self._summaryKey or 'summary')
+ self._summaryKey = None
+ _end_itunes_summary = _end_summary
+
+ def _start_enclosure(self, attrsD):
+ attrsD = self._itsAnHrefDamnIt(attrsD)
+ self._getContext().setdefault('enclosures', []).append(FeedParserDict(attrsD))
+ href = attrsD.get('href')
+ if href:
+ context = self._getContext()
+ if not context.get('id'):
+ context['id'] = href
+
+ def _start_source(self, attrsD):
+ self.insource = 1
+
+ def _end_source(self):
+ self.insource = 0
+ self._getContext()['source'] = copy.deepcopy(self.sourcedata)
+ self.sourcedata.clear()
+
+ def _start_content(self, attrsD):
+ self.pushContent('content', attrsD, 'text/plain', 1)
+ src = attrsD.get('src')
+ if src:
+ self.contentparams['src'] = src
+ self.push('content', 1)
+
+ def _start_prodlink(self, attrsD):
+ self.pushContent('content', attrsD, 'text/html', 1)
+
+ def _start_body(self, attrsD):
+ self.pushContent('content', attrsD, 'application/xhtml+xml', 1)
+ _start_xhtml_body = _start_body
+
+ def _start_content_encoded(self, attrsD):
+ self.pushContent('content', attrsD, 'text/html', 1)
+ _start_fullitem = _start_content_encoded
+
+ def _end_content(self):
+ copyToDescription = self.mapContentType(self.contentparams.get('type')) in (['text/plain'] + self.html_types)
+ value = self.popContent('content')
+ if copyToDescription:
+ self._save('description', value)
+ _end_body = _end_content
+ _end_xhtml_body = _end_content
+ _end_content_encoded = _end_content
+ _end_fullitem = _end_content
+ _end_prodlink = _end_content
+
+ def _start_itunes_image(self, attrsD):
+ self.push('itunes_image', 0)
+ self._getContext()['image'] = FeedParserDict({'href': attrsD.get('href')})
+ _start_itunes_link = _start_itunes_image
+
+ def _end_itunes_block(self):
+ value = self.pop('itunes_block', 0)
+ self._getContext()['itunes_block'] = (value == 'yes') and 1 or 0
+
+ def _end_itunes_explicit(self):
+ value = self.pop('itunes_explicit', 0)
+ self._getContext()['itunes_explicit'] = (value == 'yes') and 1 or 0
+
+if _XML_AVAILABLE:
+ class _StrictFeedParser(_FeedParserMixin, xml.sax.handler.ContentHandler):
+ def __init__(self, baseuri, baselang, encoding):
+ if _debug: sys.stderr.write('trying StrictFeedParser\n')
+ xml.sax.handler.ContentHandler.__init__(self)
+ _FeedParserMixin.__init__(self, baseuri, baselang, encoding)
+ self.bozo = 0
+ self.exc = None
+
+ def startPrefixMapping(self, prefix, uri):
+ self.trackNamespace(prefix, uri)
+
+ def startElementNS(self, name, qname, attrs):
+ namespace, localname = name
+ lowernamespace = str(namespace or '').lower()
+ if lowernamespace.find('backend.userland.com/rss') <> -1:
+ # match any backend.userland.com namespace
+ namespace = 'http://backend.userland.com/rss'
+ lowernamespace = namespace
+ if qname and qname.find(':') > 0:
+ givenprefix = qname.split(':')[0]
+ else:
+ givenprefix = None
+ prefix = self._matchnamespaces.get(lowernamespace, givenprefix)
+ if givenprefix and (prefix == None or (prefix == '' and lowernamespace == '')) and not self.namespacesInUse.has_key(givenprefix):
+ raise UndeclaredNamespace, "'%s' is not associated with a namespace" % givenprefix
+ if prefix:
+ localname = prefix + ':' + localname
+ localname = str(localname).lower()
+ if _debug: sys.stderr.write('startElementNS: qname = %s, namespace = %s, givenprefix = %s, prefix = %s, attrs = %s, localname = %s\n' % (qname, namespace, givenprefix, prefix, attrs.items(), localname))
+
+ # qname implementation is horribly broken in Python 2.1 (it
+ # doesn't report any), and slightly broken in Python 2.2 (it
+ # doesn't report the xml: namespace). So we match up namespaces
+ # with a known list first, and then possibly override them with
+ # the qnames the SAX parser gives us (if indeed it gives us any
+ # at all). Thanks to MatejC for helping me test this and
+ # tirelessly telling me that it didn't work yet.
+ attrsD = {}
+ for (namespace, attrlocalname), attrvalue in attrs._attrs.items():
+ lowernamespace = (namespace or '').lower()
+ prefix = self._matchnamespaces.get(lowernamespace, '')
+ if prefix:
+ attrlocalname = prefix + ':' + attrlocalname
+ attrsD[str(attrlocalname).lower()] = attrvalue
+ for qname in attrs.getQNames():
+ attrsD[str(qname).lower()] = attrs.getValueByQName(qname)
+ self.unknown_starttag(localname, attrsD.items())
+
+ def characters(self, text):
+ self.handle_data(text)
+
+ def endElementNS(self, name, qname):
+ namespace, localname = name
+ lowernamespace = str(namespace or '').lower()
+ if qname and qname.find(':') > 0:
+ givenprefix = qname.split(':')[0]
+ else:
+ givenprefix = ''
+ prefix = self._matchnamespaces.get(lowernamespace, givenprefix)
+ if prefix:
+ localname = prefix + ':' + localname
+ localname = str(localname).lower()
+ self.unknown_endtag(localname)
+
+ def error(self, exc):
+ self.bozo = 1
+ self.exc = exc
+
+ def fatalError(self, exc):
+ self.error(exc)
+ raise exc
+
+class _BaseHTMLProcessor(sgmllib.SGMLParser):
+ elements_no_end_tag = ['area', 'base', 'basefont', 'br', 'col', 'frame', 'hr',
+ 'img', 'input', 'isindex', 'link', 'meta', 'param']
+
+ def __init__(self, encoding):
+ self.encoding = encoding
+ if _debug: sys.stderr.write('entering BaseHTMLProcessor, encoding=%s\n' % self.encoding)
+ sgmllib.SGMLParser.__init__(self)
+
+ def reset(self):
+ self.pieces = []
+ sgmllib.SGMLParser.reset(self)
+
+ def _shorttag_replace(self, match):
+ tag = match.group(1)
+ if tag in self.elements_no_end_tag:
+ return '<' + tag + ' />'
+ else:
+ return '<' + tag + '>' + tag + '>'
+
+ def feed(self, data):
+ data = re.compile(r'', self._shorttag_replace, data) # bug [ 1399464 ] Bad regexp for _shorttag_replace
+ data = re.sub(r'<([^<\s]+?)\s*/>', self._shorttag_replace, data)
+ data = data.replace(''', "'")
+ data = data.replace('"', '"')
+ if self.encoding and type(data) == type(u''):
+ data = data.encode(self.encoding)
+ sgmllib.SGMLParser.feed(self, data)
+ sgmllib.SGMLParser.close(self)
+
+ def normalize_attrs(self, attrs):
+ # utility method to be called by descendants
+ attrs = [(k.lower(), v) for k, v in attrs]
+ attrs = [(k, k in ('rel', 'type') and v.lower() or v) for k, v in attrs]
+ return attrs
+
+ def unknown_starttag(self, tag, attrs):
+ # called for each start tag
+ # attrs is a list of (attr, value) tuples
+ # e.g. for
, tag='pre', attrs=[('class', 'screen')]
+ if _debug: sys.stderr.write('_BaseHTMLProcessor, unknown_starttag, tag=%s\n' % tag)
+ uattrs = []
+ # thanks to Kevin Marks for this breathtaking hack to deal with (valid) high-bit attribute values in UTF-8 feeds
+ for key, value in attrs:
+ if type(value) != type(u''):
+ value = unicode(value, self.encoding)
+ uattrs.append((unicode(key, self.encoding), value))
+ strattrs = u''.join([u' %s="%s"' % (key, value) for key, value in uattrs]).encode(self.encoding)
+ if tag in self.elements_no_end_tag:
+ self.pieces.append('<%(tag)s%(strattrs)s />' % locals())
+ else:
+ self.pieces.append('<%(tag)s%(strattrs)s>' % locals())
+
+ def unknown_endtag(self, tag):
+ # called for each end tag, e.g. for
+ HTMLTMPL WARNING:
+ Cannot include template: %s
+
+
+ """ % filename
+ self.DEB("CANNOT INCLUDE WARNING")
+
+ elif token == "." % token
+
+ elif DISABLE_OUTPUT not in output_control:
+ # Raw textual template data.
+ # If output of current block is not disabled, then
+ # append template data to the output buffer.
+ out += token
+
+ i += 1
+ # end of the big while loop
+
+ # Check whether all opening statements were closed.
+ if loop_name: raise TemplateError, "Missing ."
+ if output_control: raise TemplateError, "Missing or "
+ return out
+
+ ##############################################
+ # PRIVATE METHODS #
+ ##############################################
+
+ def DEB(self, str):
+ """ Print debugging message to stderr if debugging is enabled.
+ @hidden
+ """
+ if self._debug: print >> sys.stderr, str
+
+ def find_value(self, var, loop_name, loop_pass, loop_total,
+ global_override=None):
+ """ Search the self._vars data structure to find variable var
+ located in currently processed pass of a loop which
+ is currently being processed. If the variable is an ordinary
+ variable, then return it.
+
+ If the variable is an identificator of a loop, then
+ return the total number of times this loop will
+ be executed.
+
+ Return an empty string, if the variable is not
+ found at all.
+
+ @hidden
+ """
+ # Search for the requested variable in magic vars if the name
+ # of the variable starts with "__" and if we are inside a loop.
+ if self._magic_vars and var.startswith("__") and loop_name:
+ return self.magic_var(var, loop_pass[-1], loop_total[-1])
+
+ # Search for an ordinary variable or for a loop.
+ # Recursively search in self._vars for the requested variable.
+ scope = self._vars
+ globals = []
+ for i in range(len(loop_name)):
+ # If global lookup is on then push the value on the stack.
+ if ((self._global_vars and global_override != "0") or \
+ global_override == "1") and scope.has_key(var) and \
+ self.is_ordinary_var(scope[var]):
+ globals.append(scope[var])
+
+ # Descent deeper into the hierarchy.
+ if scope.has_key(loop_name[i]) and scope[loop_name[i]]:
+ scope = scope[loop_name[i]][loop_pass[i]]
+ else:
+ return ""
+
+ if scope.has_key(var):
+ # Value exists in current loop.
+ if type(scope[var]) == ListType:
+ # The requested value is a loop.
+ # Return total number of its passes.
+ return len(scope[var])
+ else:
+ return scope[var]
+ elif globals and \
+ ((self._global_vars and global_override != "0") or \
+ global_override == "1"):
+ # Return globally looked up value.
+ return globals.pop()
+ else:
+ # No value found.
+ if var[0].isupper():
+ # This is a loop name.
+ # Return zero, because the user wants to know number
+ # of its passes.
+ return 0
+ else:
+ return ""
+
+ def magic_var(self, var, loop_pass, loop_total):
+ """ Resolve and return value of a magic variable.
+ Raise an exception if the magic variable is not recognized.
+
+ @hidden
+ """
+ self.DEB("MAGIC: '%s', PASS: %d, TOTAL: %d"\
+ % (var, loop_pass, loop_total))
+ if var == "__FIRST__":
+ if loop_pass == 0:
+ return 1
+ else:
+ return 0
+ elif var == "__LAST__":
+ if loop_pass == loop_total - 1:
+ return 1
+ else:
+ return 0
+ elif var == "__INNER__":
+ # If this is neither the first nor the last pass.
+ if loop_pass != 0 and loop_pass != loop_total - 1:
+ return 1
+ else:
+ return 0
+ elif var == "__PASS__":
+ # Magic variable __PASS__ counts passes from one.
+ return loop_pass + 1
+ elif var == "__PASSTOTAL__":
+ return loop_total
+ elif var == "__ODD__":
+ # Internally pass numbers stored in loop_pass are counted from
+ # zero. But the template language presents them counted from one.
+ # Therefore we must add one to the actual loop_pass value to get
+ # the value we present to the user.
+ if (loop_pass + 1) % 2 != 0:
+ return 1
+ else:
+ return 0
+ elif var.startswith("__EVERY__"):
+ # Magic variable __EVERY__x is never true in first or last pass.
+ if loop_pass != 0 and loop_pass != loop_total - 1:
+ # Check if an integer follows the variable name.
+ try:
+ every = int(var[9:]) # nine is length of "__EVERY__"
+ except ValueError:
+ raise TemplateError, "Magic variable __EVERY__x: "\
+ "Invalid pass number."
+ else:
+ if not every:
+ raise TemplateError, "Magic variable __EVERY__x: "\
+ "Pass number cannot be zero."
+ elif (loop_pass + 1) % every == 0:
+ self.DEB("MAGIC: EVERY: " + str(every))
+ return 1
+ else:
+ return 0
+ else:
+ return 0
+ else:
+ raise TemplateError, "Invalid magic variable '%s'." % var
+
+ def escape(self, str, override=""):
+ """ Escape a string either by HTML escaping or by URL escaping.
+ @hidden
+ """
+ ESCAPE_QUOTES = 1
+ if (self._html_escape and override != "NONE" and override != "0" and \
+ override != "URL") or override == "HTML" or override == "1":
+ return cgi.escape(str, ESCAPE_QUOTES)
+ elif override == "URL":
+ return urllib.quote_plus(str)
+ else:
+ return str
+
+ def is_ordinary_var(self, var):
+ """ Return true if var is a scalar. (not a reference to loop)
+ @hidden
+ """
+ if type(var) == StringType or type(var) == IntType or \
+ type(var) == LongType or type(var) == FloatType:
+ return 1
+ else:
+ return 0
+
+
+##############################################
+# CLASS: TemplateCompiler #
+##############################################
+
+class TemplateCompiler:
+ """ Preprocess, parse, tokenize and compile the template.
+
+ This class parses the template and produces a 'compiled' form
+ of it. This compiled form is an instance of the Template
+ class. The compiled form is used as input for the TemplateProcessor
+ which uses it to actually process the template.
+
+ This class should be used direcly only when you need to compile
+ a template from a string. If your template is in a file, then you
+ should use the TemplateManager class which provides
+ a higher level interface to this class and also can save the
+ compiled template to disk in a precompiled form.
+ """
+
+ def __init__(self, include=1, max_include=5, comments=1, gettext=0,
+ debug=0):
+ """ Constructor.
+
+ @header __init__(include=1, max_include=5, comments=1, gettext=0,
+ debug=0)
+
+ @param include Enable or disable included templates.
+ @param max_include Maximum depth of nested inclusions.
+ @param comments Enable or disable template comments.
+ @param gettext Enable or disable gettext support.
+ @param debug Enable or disable debugging messages.
+ """
+
+ self._include = include
+ self._max_include = max_include
+ self._comments = comments
+ self._gettext = gettext
+ self._debug = debug
+
+ # This is a list of filenames of all included templates.
+ # It's modified by the include_templates() method.
+ self._include_files = []
+
+ # This is a counter of current inclusion depth. It's used to prevent
+ # infinite recursive includes.
+ self._include_level = 0
+
+ def compile(self, file):
+ """ Compile template from a file.
+
+ @header compile(file)
+ @return Compiled template.
+ The return value is an instance of the Template
+ class.
+
+ @param file Filename of the template.
+ See the prepare() method of the TemplateManager
+ class for exaplanation of this parameter.
+ """
+
+ self.DEB("COMPILING FROM FILE: " + file)
+ self._include_path = os.path.join(os.path.dirname(file), INCLUDE_DIR)
+ tokens = self.parse(self.read(file))
+ compile_params = (self._include, self._max_include, self._comments,
+ self._gettext)
+ return Template(__version__, file, self._include_files,
+ tokens, compile_params, self._debug)
+
+ def compile_string(self, data):
+ """ Compile template from a string.
+
+ This method compiles a template from a string. The
+ template cannot include any templates.
+ TMPL_INCLUDE statements are turned into warnings.
+
+ @header compile_string(data)
+ @return Compiled template.
+ The return value is an instance of the Template
+ class.
+
+ @param data String containing the template data.
+ """
+ self.DEB("COMPILING FROM STRING")
+ self._include = 0
+ tokens = self.parse(data)
+ compile_params = (self._include, self._max_include, self._comments,
+ self._gettext)
+ return Template(__version__, None, None, tokens, compile_params,
+ self._debug)
+
+ ##############################################
+ # PRIVATE METHODS #
+ ##############################################
+
+ def DEB(self, str):
+ """ Print debugging message to stderr if debugging is enabled.
+ @hidden
+ """
+ if self._debug: print >> sys.stderr, str
+
+ def read(self, filename):
+ """ Read content of file and return it. Raise an error if a problem
+ occurs.
+ @hidden
+ """
+ self.DEB("READING: " + filename)
+ try:
+ f = None
+ try:
+ f = open(filename, "r")
+ data = f.read()
+ except IOError, (errno, errstr):
+ raise TemplateError, "IO error while reading template '%s': "\
+ "(%d) %s" % (filename, errno, errstr)
+ else:
+ return data
+ finally:
+ if f: f.close()
+
+ def parse(self, template_data):
+ """ Parse the template. This method is recursively called from
+ within the include_templates() method.
+
+ @return List of processing tokens.
+ @hidden
+ """
+ if self._comments:
+ self.DEB("PREPROCESS: COMMENTS")
+ template_data = self.remove_comments(template_data)
+ tokens = self.tokenize(template_data)
+ if self._include:
+ self.DEB("PREPROCESS: INCLUDES")
+ self.include_templates(tokens)
+ return tokens
+
+ def remove_comments(self, template_data):
+ """ Remove comments from the template data.
+ @hidden
+ """
+ pattern = r"### .*"
+ return re.sub(pattern, "", template_data)
+
+ def include_templates(self, tokens):
+ """ Process TMPL_INCLUDE statements. Use the include_level counter
+ to prevent infinite recursion. Record paths to all included
+ templates to self._include_files.
+ @hidden
+ """
+ i = 0
+ out = "" # buffer for output
+ skip_params = 0
+
+ # Process the list of tokens.
+ while 1:
+ if i == len(tokens): break
+ if skip_params:
+ skip_params = 0
+ i += PARAMS_NUMBER
+ continue
+
+ token = tokens[i]
+ if token == "."
+ self._include_level += 1
+ if self._include_level > self._max_include:
+ # Do not include the template.
+ # Protection against infinite recursive includes.
+ skip_params = 1
+ self.DEB("INCLUDE: LIMIT REACHED: " + filename)
+ else:
+ # Include the template.
+ skip_params = 0
+ include_file = os.path.join(self._include_path, filename)
+ self._include_files.append(include_file)
+ include_data = self.read(include_file)
+ include_tokens = self.parse(include_data)
+
+ # Append the tokens from the included template to actual
+ # position in the tokens list, replacing the TMPL_INCLUDE
+ # token and its parameters.
+ tokens[i:i+PARAMS_NUMBER+1] = include_tokens
+ i = i + len(include_tokens)
+ self.DEB("INCLUDED: " + filename)
+ continue # Do not increment 'i' below.
+ i += 1
+ # end of the main while loop
+
+ if self._include_level > 0: self._include_level -= 1
+ return out
+
+ def tokenize(self, template_data):
+ """ Split the template into tokens separated by template statements.
+ The statements itself and associated parameters are also
+ separately included in the resulting list of tokens.
+ Return list of the tokens.
+
+ @hidden
+ """
+ self.DEB("TOKENIZING TEMPLATE")
+ # NOTE: The TWO double quotes in character class in the regexp below
+ # are there only to prevent confusion of syntax highlighter in Emacs.
+ pattern = r"""
+ (?:^[ \t]+)? # eat spaces, tabs (opt.)
+ (<
+ (?:!--[ ])? # comment start + space (opt.)
+ /?TMPL_[A-Z]+ # closing slash / (opt.) + statement
+ [ a-zA-Z0-9""/.=:_\\-]* # this spans also comments ending (--)
+ >)
+ [%s]? # eat trailing newline (opt.)
+ """ % os.linesep
+ rc = re.compile(pattern, re.VERBOSE | re.MULTILINE)
+ split = rc.split(template_data)
+ tokens = []
+ for statement in split:
+ if statement.startswith(" 0 and '=' not in params[0]:
+ # implicit identifier
+ name = params[0]
+ del params[0]
+ else:
+ # explicit identifier as a 'NAME' parameter
+ name = self.find_param("NAME", params)
+ self.DEB("TOKENIZER: NAME: " + str(name))
+ return name
+
+ def find_param(self, param, params):
+ """ Extract value of parameter from a statement.
+ @hidden
+ """
+ for pair in params:
+ name, value = pair.split("=")
+ if not name or not value:
+ raise TemplateError, "Syntax error in template."
+ if name == param:
+ if value[0] == '"':
+ # The value is in double quotes.
+ ret_value = value[1:-1]
+ else:
+ # The value is without double quotes.
+ ret_value = value
+ self.DEB("TOKENIZER: PARAM: '%s' => '%s'" % (param, ret_value))
+ return ret_value
+ else:
+ self.DEB("TOKENIZER: PARAM: '%s' => NOT DEFINED" % param)
+ return None
+
+
+##############################################
+# CLASS: Template #
+##############################################
+
+class Template:
+ """ This class represents a compiled template.
+
+ This class provides storage and methods for the compiled template
+ and associated metadata. It's serialized by pickle if we need to
+ save the compiled template to disk in a precompiled form.
+
+ You should never instantiate this class directly. Always use the
+ TemplateManager or TemplateCompiler classes to
+ create the instances of this class.
+
+ The only method which you can directly use is the is_uptodate
+ method.
+ """
+
+ def __init__(self, version, file, include_files, tokens, compile_params,
+ debug=0):
+ """ Constructor.
+ @hidden
+ """
+ self._version = version
+ self._file = file
+ self._tokens = tokens
+ self._compile_params = compile_params
+ self._debug = debug
+ self._mtime = None
+ self._include_mtimes = {}
+
+ if not file:
+ self.DEB("TEMPLATE WAS COMPILED FROM A STRING")
+ return
+
+ # Save modifitcation time of the main template file.
+ if os.path.isfile(file):
+ self._mtime = os.path.getmtime(file)
+ else:
+ raise TemplateError, "Template: file does not exist: '%s'" % file
+
+ # Save modificaton times of all included template files.
+ for inc_file in include_files:
+ if os.path.isfile(inc_file):
+ self._include_mtimes[inc_file] = os.path.getmtime(inc_file)
+ else:
+ raise TemplateError, "Template: file does not exist: '%s'"\
+ % inc_file
+
+ self.DEB("NEW TEMPLATE CREATED")
+
+ def is_uptodate(self, compile_params=None):
+ """ Check whether the compiled template is uptodate.
+
+ Return true if this compiled template is uptodate.
+ Return false, if the template source file was changed on the
+ disk since it was compiled.
+ Works by comparison of modification times.
+ Also takes modification times of all included templates
+ into account.
+
+ @header is_uptodate(compile_params=None)
+ @return True if the template is uptodate, false otherwise.
+
+ @param compile_params Only for internal use.
+ Do not use this optional parameter. It's intended only for
+ internal use by the TemplateManager.
+ """
+ if not self._file:
+ self.DEB("TEMPLATE COMPILED FROM A STRING")
+ return 0
+
+ if self._version != __version__:
+ self.DEB("TEMPLATE: VERSION NOT UPTODATE")
+ return 0
+
+ if compile_params != None and compile_params != self._compile_params:
+ self.DEB("TEMPLATE: DIFFERENT COMPILATION PARAMS")
+ return 0
+
+ # Check modification times of the main template and all included
+ # templates. If the included template no longer exists, then
+ # the problem will be resolved when the template is recompiled.
+
+ # Main template file.
+ if not (os.path.isfile(self._file) and \
+ self._mtime == os.path.getmtime(self._file)):
+ self.DEB("TEMPLATE: NOT UPTODATE: " + self._file)
+ return 0
+
+ # Included templates.
+ for inc_file in self._include_mtimes.keys():
+ if not (os.path.isfile(inc_file) and \
+ self._include_mtimes[inc_file] == \
+ os.path.getmtime(inc_file)):
+ self.DEB("TEMPLATE: NOT UPTODATE: " + inc_file)
+ return 0
+ else:
+ self.DEB("TEMPLATE: UPTODATE")
+ return 1
+
+ def tokens(self):
+ """ Get tokens of this template.
+ @hidden
+ """
+ return self._tokens
+
+ def file(self):
+ """ Get filename of the main file of this template.
+ @hidden
+ """
+ return self._file
+
+ def debug(self, debug):
+ """ Get debugging state.
+ @hidden
+ """
+ self._debug = debug
+
+ ##############################################
+ # PRIVATE METHODS #
+ ##############################################
+
+ def __getstate__(self):
+ """ Used by pickle when the class is serialized.
+ Remove the 'debug' attribute before serialization.
+ @hidden
+ """
+ dict = copy.copy(self.__dict__)
+ del dict["_debug"]
+ return dict
+
+ def __setstate__(self, dict):
+ """ Used by pickle when the class is unserialized.
+ Add the 'debug' attribute.
+ @hidden
+ """
+ dict["_debug"] = 0
+ self.__dict__ = dict
+
+
+ def DEB(self, str):
+ """ Print debugging message to stderr.
+ @hidden
+ """
+ if self._debug: print >> sys.stderr, str
+
+
+##############################################
+# EXCEPTIONS #
+##############################################
+
+class TemplateError(Exception):
+ """ Fatal exception. Raised on runtime or template syntax errors.
+
+ This exception is raised when a runtime error occurs or when a syntax
+ error in the template is found. It has one parameter which always
+ is a string containing a description of the error.
+
+ All potential IOError exceptions are handled by the module and are
+ converted to TemplateError exceptions. That means you should catch the
+ TemplateError exception if there is a possibility that for example
+ the template file will not be accesssible.
+
+ The exception can be raised by constructors or by any method of any
+ class.
+
+ The instance is no longer usable when this exception is raised.
+ """
+
+ def __init__(self, error):
+ """ Constructor.
+ @hidden
+ """
+ Exception.__init__(self, "Htmltmpl error: " + error)
+
+
+class PrecompiledError(Exception):
+ """ This exception is _PRIVATE_ and non fatal.
+ @hidden
+ """
+
+ def __init__(self, template):
+ """ Constructor.
+ @hidden
+ """
+ Exception.__init__(self, template)
+
diff --git a/code/planet/sanitize.py b/code/planet/sanitize.py
new file mode 100644
index 0000000..c98b14d
--- /dev/null
+++ b/code/planet/sanitize.py
@@ -0,0 +1,354 @@
+"""
+sanitize: bringing sanitiy to world of messed-up data
+"""
+
+__author__ = ["Mark Pilgrim ",
+ "Aaron Swartz "]
+__contributors__ = ["Sam Ruby "]
+__license__ = "BSD"
+__version__ = "0.25"
+
+_debug = 0
+
+# If you want sanitize to automatically run HTML markup through HTML Tidy, set
+# this to 1. Requires mxTidy
+# or utidylib .
+TIDY_MARKUP = 0
+
+# List of Python interfaces for HTML Tidy, in order of preference. Only useful
+# if TIDY_MARKUP = 1
+PREFERRED_TIDY_INTERFACES = ["uTidy", "mxTidy"]
+
+import sgmllib, re
+
+# chardet library auto-detects character encodings
+# Download from http://chardet.feedparser.org/
+try:
+ import chardet
+ if _debug:
+ import chardet.constants
+ chardet.constants._debug = 1
+
+ _chardet = lambda data: chardet.detect(data)['encoding']
+except:
+ chardet = None
+ _chardet = lambda data: None
+
+class _BaseHTMLProcessor(sgmllib.SGMLParser):
+ elements_no_end_tag = ['area', 'base', 'basefont', 'br', 'col', 'frame', 'hr',
+ 'img', 'input', 'isindex', 'link', 'meta', 'param']
+
+ _r_barebang = re.compile(r'')
+
+ def __init__(self, encoding):
+ self.encoding = encoding
+ if _debug: sys.stderr.write('entering BaseHTMLProcessor, encoding=%s\n' % self.encoding)
+ sgmllib.SGMLParser.__init__(self)
+
+ def reset(self):
+ self.pieces = []
+ sgmllib.SGMLParser.reset(self)
+
+ def _shorttag_replace(self, match):
+ tag = match.group(1)
+ if tag in self.elements_no_end_tag:
+ return '<' + tag + ' />'
+ else:
+ return '<' + tag + '>' + tag + '>'
+
+ def feed(self, data):
+ data = self._r_barebang.sub(r'<!\1', data)
+ data = self._r_bareamp.sub("&", data)
+ data = self._r_shorttag.sub(self._shorttag_replace, data)
+ if self.encoding and type(data) == type(u''):
+ data = data.encode(self.encoding)
+ sgmllib.SGMLParser.feed(self, data)
+
+ def normalize_attrs(self, attrs):
+ # utility method to be called by descendants
+ attrs = [(k.lower(), v) for k, v in attrs]
+ attrs = [(k, k in ('rel', 'type') and v.lower() or v) for k, v in attrs]
+ return attrs
+
+ def unknown_starttag(self, tag, attrs):
+ # called for each start tag
+ # attrs is a list of (attr, value) tuples
+ # e.g. for
, tag='pre', attrs=[('class', 'screen')]
+ if _debug: sys.stderr.write('_BaseHTMLProcessor, unknown_starttag, tag=%s\n' % tag)
+ uattrs = []
+ # thanks to Kevin Marks for this breathtaking hack to deal with (valid) high-bit attribute values in UTF-8 feeds
+ for key, value in attrs:
+ if type(value) != type(u''):
+ value = unicode(value, self.encoding)
+ uattrs.append((unicode(key, self.encoding), value))
+ strattrs = u''.join([u' %s="%s"' % (key, value) for key, value in uattrs]).encode(self.encoding)
+ if tag in self.elements_no_end_tag:
+ self.pieces.append('<%(tag)s%(strattrs)s />' % locals())
+ else:
+ self.pieces.append('<%(tag)s%(strattrs)s>' % locals())
+
+ def unknown_endtag(self, tag):
+ # called for each end tag, e.g. for
+
+### One of the two loops available is the Channels loop. This allows you
+### to easily create a list of subscriptions, which is exactly what we'll do
+### here.
+
+### Note that we can also expand variables inside HTML tags, but we need
+### to be cautious and HTML-escape any illegal characters using the form
+###
+
+
+
+### The other loop is the Items loop, which will get iterated for each
+### news item.
+
+
+
+### Visually distinguish articles from different days by checking for
+### the new_date flag. This demonstrates the ...
+### check.
+
+
+
+
+
+### Group consecutive articles by the same author together by checking
+### for the new_channel flag.
+
+
+