Improve scripting and its experience in Scribus: Difference between revisions

From Scribus Wiki
Jump to navigation Jump to search
m (Protected "Improve scripting and its experience in Scribus": Archive ([edit=sysop] (indefinite) [move=sysop] (indefinite)))
 
(15 intermediate revisions by 5 users not shown)
Line 1: Line 1:
[[Category:GSOC]] [[Category:GSOC 2007]] {{Archive}}
== Introduction ==
== Introduction ==


Line 6: Line 8:
* automate repetitive tasks to improve usability and to save time
* automate repetitive tasks to improve usability and to save time
* test ideas  - Python is great for prototyping and sometimes it makes more sense to improve the prototype instead of rewriting everything in C++
* test ideas  - Python is great for prototyping and sometimes it makes more sense to improve the prototype instead of rewriting everything in C++
 
* use functionality that does not exist or would introduce dirty dependencies like for example using PyUNO to access OpenOffice.org (it's even possible to run OO.org in the same thread the qapp runs in) or reportlab.org which allows to create easy custom PDF's using some XML-templates or whatever...


I actually have written prototypes (among others) of
I actually have written prototypes (among others) of
Line 14: Line 16:
* connectors to databases and content management systems
* connectors to databases and content management systems
* GUI hacks (full screen mode, tear-off menus etc.)
* GUI hacks (full screen mode, tear-off menus etc.)


== Current problems ==
== Current problems ==


As you see in the current state scripter is already powerful but it lacks
As you see in the current state scripter is already powerful but it lacks
* full integration into the GUI and the core: Scripts cannot easily add menu entries or toolbar icons, save settings into the Scribus configuration file, etc.  [CR: This will probably need a decently designed public C++ API for Scribus, plus a SIP wrapper for it]
* full integration into the GUI and the core: Scripts cannot easily add menu entries or toolbar icons, save settings into the Scribus configuration file, etc.  [CR: This will probably need a decently designed [[public C++ API]] for Scribus, plus a SIP wrapper for it]
* completeness: While writing scripts you sometimes find things which are still missing. For example there is a function to zoom the document view. But there lacks a function to get the current zoom factor. [CR: Again, need a real public API in C++ to make any kind of completeness effort viable and maintainable]
* completeness: While writing scripts you sometimes find things which are still missing. For example there is a function to zoom the document view. But there lacks a function to get the current zoom factor. [CR: Again, need a real public API in C++ to make any kind of completeness effort viable and maintainable]
* ease of use: The script console is really minimalistic and needs improvement.  Also there is no standard way to distribute and install scripts.  
* ease of use: The script console is really minimalistic and needs improvement.  Also there is no standard way to distribute and install scripts.  
* security: Scripter should warn you if a script wants to import a module which can access the file system.  The user should be asked if he trusts a script. [CR: unfortunately the first part of this is not viable, though the second is - see below]
* security: Scripter should warn you if a script wants to import a module which can access the file system.  The user should be asked if he trusts a script. [CR: unfortunately the first part of this is not viable, though the second is - see [[Scripter Security]] ]
 
== Security ==
 
Python has absolutely no way to ensure that code can be "checked" a-la perl restricted mode or Java's bytecode security model. The restricted python mode was proven to be ineffective and has been removed. Every suggestion anybody has ever come up with as to how to restrict Python programs with the CPython interpreter has been proven to be breakable (see innumerable discussions on the newsgroup - I won't repeat the whole large topic here). Under these circumstances we would be instilling a false sense of security by offering any sort of restricted mode to users.
 
If you want a secure language for automation, embed Lua, (maybe) Qt Script for Applications (an ECMAScript/JavaScript implementation), or even restricted mode Java. Python is not suitable.
 
What we need to do is ensure that documents either (a) can NEVER embed scripts, or (b) can embed scripts but ONLY run them if the user specifically turns that on in the prefs AND accepts a prompt when opening the document. We might use a signing scheme to permit the prompt to be disabled for trusted script publishers as nominated by the user. This prevents the document virus problem we see in MS Office.
 
Personally I don't think we should permit script embedding in documents at all. Make the user install the script and understand that by doing so they give it the same rights as any other program they install on the system. If we want to support script embedding, we're better off choosing a more restricted language for that and leaving Python for extending the app and for advanced scripting with GUIs etc.
 
=== Answer to CR===
 
I am aware of the [http://www.amk.ca/python/howto/rexec/rexec.html problems with restricted execution] and I also believe there is no ideal solution. But I think the security issue needs to be addressed nevertheless and a little more security is better than no security. In most cases insecure scripts are just bad code containing errors which might harm the system and are no trojans. So:
* Warn the user that an unsigned script cannot be trusted and that he runs it on his own risk and that the script can do everything the current user can do.  (Most users are not aware that this is possible at all).  [CR: Agreed]
* Use import hooks and tell the user that a script e.g. accesses the "os" or "posix" module. [CR: Couldn't disagree more. See the security notes.]
* Wrap builtins like "open". There are ways to circumvent the wrapped functions (see link below) but only trojans do this and it would a least help for simple scripts. [CR: As per last. ]
* Make a repository of trusted scripts (needs some kind of skilled moderators). [CR: This seems like an excellent idea to me, and would fit in well with a signing scheme.]
* Evaluate [http://mail.python.org/pipermail/python-dev/2003-August/037791.html RestrictedPython from Zope] which  proves to work in Zope and is not related to the old rpython/Bastion modules.
 
=== Reply (CR) ===
 
I very strongly disagree with the idea of trying to use import hooks and builtin wrappers to provide any sort of access control - or even for information purposes. Such techniques are easily bypassed by malicious users, and more importantly will impart a false sense of security to users. Furthermore, if we have an effective restricted execution environment such measures will be unnecessary, since programs *can't* access the file system etc.
 
RestrictedPython looks more interesting, in that it takes a saner "ban it until it's proven safe" approach. I'm still not overly convinced, though, and will want to see some strong evidence it's solid. Preferably a BFDL endorsement ;-) . In short, I don't have hard evidence against it yet but past experience with people's attempts to create restricted Python runtimes makes me dubious. The last thing we want is anything that makes users think running scripts from Joe Random is safe if we can't absolutely guarantee that's the case.
 
In particular, I think that combining the idea of restricted execution with any use of SIP or PyQt is raving lunacy... somewhat akin to giving a loaded handgun to a toddler, but saying it's safe because the safety catch is on. PyQt in particular gives the programmer pretty much free reign to do whatever they want.
 
The Scribus team has long wanted to get rid of the current Python interface completely, and replace it with a scratch rewrite. To do this, we've been planning on writing a pure C++ wrapper around Scribus's internal implementation. This interface will be available for plugins to use directly, and will be friendly for wrapper generators like SIP. That means that a Python interface will have similar capabilities to what plugins can do (especially if we use SIP and PyQt). It'd save a bunch of work, produce a much cleaner and massively more functional Python interface - and it'll do so without all the hideous hand-coded functions we currently have. Users could use Python's database interface modules and all its other functionality to create very powerful extensions to Scribus, including whole new dialogs etc (with PyQt) and extensions to select parts of the current Scribus GUI*. They'd also be able to make use of hooks to listen on certain events in the Scribus core and (maybe) redirect or block them. By using SIP to wrap the C++ interface we can actually do a lot of this quite easily.
 
The downside of the above is that making it secure is almost certainly wishful thinking.
 
I think a limited and more secure scripting interface is a '''completely different problem''' to providing a more complete and powerful scripting interface for Python. These two functions really couldn't be more different, and I honestly think we'd be better off providing totally different interfaces for them. Perhaps this can be done with Python too (in the _same_ interpreter instance as the unrestricted mode, remember) ... but I'm still not convinced. I remain of the view that such a restricted interface would be much better done using something like lua+SWIG or Qt Script for Applications. Both lua and QSA are built on more of a "you can't do it unless the developer exposes it" model rather than a completely open model like Python.
 
Another issue you're probably not familiar with is that of sub-interpreters. Python's sub-intepreter feature (used to provide isolation of different scripts) is not really designed for single-threaded use in an app like Scribus. It's more for things like mod_python in servers. Sub interpreters are causing us a lot of pain in the scripter at the moment, and do not look like they'll ever get improved in Python so they'll fit our needs. Things like PyQt simply do not work within subinterpreters. However, if we drop the use of sub interpreters, scripts will need to be very aware of the shared __main__ namespace, shared module state, etc, and will need to be written more carefully. Currently we support both - the user can run a script in the top level interpreter (for PyQt use etc) or in a subinterpreter. This causes its own problems.
 
I think that's another reason to pick a different tool that's designed from the beginning for the restricted scripting uses we want. Yes, this does mean that there'd be two different scripting languages - a "power scripting and extension writing" environment and an "application automation" environment. Essentially, Python scripts would become plug-ins, with a more suitable language used for basic scripting tasks. In a way I think that's a good thing, especially if we used JavaScript (QSA). Many apps, such as the Adobe suite, use JavaScript/ECMAScript for automation and scripting, so this fits in much better with common practice. In the OSS world, Perl and LISP-variant languages are more common for embedding ... and I for one wouldn't inflict either of those on a Scribus user.
 
'''I would like to stress at this point that I am EXTREMELY happy that you're interested in the scripting interface and in working on it.''' I really liked the work you did before on the extension manager and your OpenClipArt work, and if you remember I spent some time helping to get PyQt to actually work in Scribus so they'd be more viable. My only problem is with the security issue... and I think that should simply be left wide open for Python scripts, with some non-scripting safeguards such as script signing and the total inability to embed Python scripts in documents.


== How to make scripting situation better ==
== How to make scripting situation better ==
Line 68: Line 30:


=== For the user ===
=== For the user ===
* a standard way to find and to install extensions and scripts from the web, preferably from a trusted repository  
* a standard way to find and to install extensions and scripts from the web, preferably from a trusted repository [CR: Great idea, especially if combined with signing of scripts for integrity etc]
* scripts look and behave as native parts of Scribus
* scripts look and behave as native parts of Scribus
=== For the developer ===
=== For the developer ===
* specifications on how scripts can be integrated with Scribus
* specifications on how scripts can be integrated with Scribus
Line 84: Line 47:
Mozilla Firefox shows what is possible with easily installable extensions. People without knowledge of C-programming can realize crazy ideas and useful features. A lot of the popularity of Firefox comes from its add-ons.
Mozilla Firefox shows what is possible with easily installable extensions. People without knowledge of C-programming can realize crazy ideas and useful features. A lot of the popularity of Firefox comes from its add-ons.


== Discussion ==
[CR: discussion moved to [[Talk:Improve_scripting_and_its_experience_in_Scribus|The talk page]] for this article.]
 
[CR: The very first step we need is a sane internal API that we can expose for scripts to use - including SIP wrappers for parts of the GUI that we want to open up for manipulation, etc. Discovery through Qt's parent/child mechanism is limited and clumsy. With a decent C++ API and a SIP wrapper we can start making more sane, powerful scripts - and can probably integrate script management into the existing plugin manager.]
 
''Answer to CR'': Of course it would be great to have direct access to all objects and their methods. And this would be a great final goal. Currently with the introspection features of Qt (children, queryList, allWidgets, ..) you can only get the base classes, so you get QMainWindow and not ScribusMainWindow. Wrapping everything with SIP is a large task and I think it should be done bit by bit when something is needed. I am currently looking what needs to be done to wrap Scribus classes with sip.
As an interim solution it would already helpful for finding an widget if most widgets in Scribus would have an internal name which can be set with QObject.setName or with the constructor.
Then already a lot is possible. Hooking into the GUI, responding to signals etc can be done.
 
''CR: reply'': Full access to every internal Scribus method really isn't ever going to be realistic, nor would it be useful. There's a lot of internal functionality in Scribus that neither scripts nor plug-ins should ever see or be able to see. Much of it also changes quite quickly, so exposing it would cause major compatibility problems. For example, a plugin or script should not be aware of, or have to care about, the details of how text is represented in Scribus's internal data structures ... it should just get access to a somewhat higher level API for text manipulation. What I suggest is that an important initial step before any sort of serious effort to expose more of Scribus to scripts is to provide a sane C++ API that hides much of these internal details behind more stable and somewhat simplified interfaces. These interfaces could then be wrapped using SIP for use by scripts and plug-ins.
 
That, in my opinion, would be a really GREAT GSoC project. I also think you have some great ideas with regards to making scripts easier to find, providing sensible and consistent ways to install them, etc. Don't let me discourage you ... I'm not trying to, I'm just trying to point out potential issues with the ''approach'' rather than goals.
 
That's not to say that permitting full introspection via PyQt (tagging everything with object names) is inherently a bad idea, but it will lead to scripts that'll only work for one particular version of Scribus. They'll sometimes break even on patch releases. It's useful as an interim step, I agree, but it's not something I'd ever seriously rely on. Wrapping Scribus's internal objects with SIP isn't much better, either, since they're hardly API stable. IMO we _REALLY_ need a more stable C++ API to expose Scribus's functionality to scripts and plug-ins. I spent quite a long time maintaining the scripter alongside Petr, so I am all to familiar with how much breakage there is from version to version and how hard it can be to keep up.
 
Some articles on the wiki that may be of note: [[Known_Scripter_Issues]], [[Experimental_PyQt_projects]] and [[Extension_script_discussion]] .


== Implementation details ==
== Implementation details ==
Line 107: Line 56:
* functions to get the configuration directory and the program directory
* functions to get the configuration directory and the program directory
* some widgets need to have a name so PyQt can find and access them  
* some widgets need to have a name so PyQt can find and access them  
* Bindings for the preferences manager [CR]
* Bindings for the menu manager [CR]
* (more to come soon)
* (more to come soon)


Line 117: Line 68:
* [http://henning.cco-ev.de/scribus Example for a repository]
* [http://henning.cco-ev.de/scribus Example for a repository]
* [http://bugs.scribus.net/view.php?id=2160 I once wrote a patch which added the importSVG-function to scripter]
* [http://bugs.scribus.net/view.php?id=2160 I once wrote a patch which added the importSVG-function to scripter]
----
= Excerpt from submitted abstract and application to GSoC 2007 =
== Rationale ==
Scribus shows that serious desktop publishing is possible with free software. Scribus has many powerful features which are commonly mentioned in articles, e.g. the high quality PDF export and the color management.
But the potential of the scripting interface is mostly underestimated. The Scripter plug-in is included in the Scribus main distribution together with some example scripts. However it does not get the attention it deserves in my point of view. 
Because of that I want to show what is possible with the Python scripting interface in Scribus.
== Overview ==
Other open-source projects like Firefox, Blender, Inkscape, OpenOffice or Gimp have shown that bindings to scripting languages result in many new features and contribute largely to the success of the application. People without knowledge of C/C++ but some Python-, JavaScript- or Perl-skills can easily write import-/export-filters, create wizards, write plug-ins or realize something nobody else has thought about before. The distributed scripts can be used without recompiling and even without knowledge of the scripting language.
Scripts can help to automate tasks and can fit the special needs of a user. Also it is possible to test new ideas. Languages like Python are great for prototyping. And sometimes it makes more sense to stay with Python instead of rewriting everything in a compiled language.
As Scribus already provides a good basis with its Scripter plug-in I want to improve it and its usability. To show the power of the scripting interface I want to write some small and some larger example scripts (resp. extension scripts).  My idea is that Scribus needs a script-/extension-manager like Firefox has for themes and add-ons. Like Firefox a user can browse the scripts on a website and install them by just clicking a link. So my task will also include building a small web application to provide a repository of scripts.
So the project consists of three tasks which are mutually dependent.
1. The script-manager based on PyQt and its QTextBrowser
2. The example scripts and extensions. Planed are an OpenClipArt browser and a simple spell-checker.
3. The repository. Either a CGI or a script which generates static pages
At each task I will determine which functionally is missing and how these sub-projects can be linked together. This will show what needs to be done in the scripting interface.  Currently scripts do not look like native Scribus components because e.g. they cannot be connected to menu-/toolbar-entries or context-menus. There is also no standard way to install scripts to make them automatically available. These things need to be addressed.
Some implementation issues will be discussed with the Scribus team, e.g. about security (already partly done), integration or future directions.
Currently there are thoughts in the Scribus team about creating new automatic bindings. Because of that I will do two things:
1. I will try to keep my code modular and flexible. It should be relatively easy to port the script-/extension-manager to the new bindings when available.
2. I will evaluate different binding generators (sip, swig, boost, smoke) and make tests to see which will be the best path to go (probably sip with a separate C++ API)
== Benefits ==
1. Firefox showed that people liked the flexibility to install add-ons. My project will bring this feature to Scribus  in the hope that people will enjoy Scribus even more and it will attract new users.
2. The script interface and its experience will improve which will hopefully attract new script writers.
3. Scribus gets a repository for scripts anyone can easily access. I would be willing to host it after the GSoC.
4. Some C++ code is not needed anymore and could probably be removed, e.g. the script console. I already wrote a better replacement in PyQt.
== Deliverables ==
* a script manager
* example scripts
* a script repository
* documentation for the script writers

Latest revision as of 18:13, 5 January 2011

Archive icon.png
 This article is ARCHIVED. It is a part of the Scribus Community's history and is kept here for reference and as a representation of the contributed efforts. It is protected from editing, but its source can be copied.

Introduction

The potential of the scripting interface seems highly underestimated. You can

  • add features without knowing C++ and without recompiling
  • make something which directly fits your own needs
  • automate repetitive tasks to improve usability and to save time
  • test ideas - Python is great for prototyping and sometimes it makes more sense to improve the prototype instead of rewriting everything in C++
  • use functionality that does not exist or would introduce dirty dependencies like for example using PyUNO to access OpenOffice.org (it's even possible to run OO.org in the same thread the qapp runs in) or reportlab.org which allows to create easy custom PDF's using some XML-templates or whatever...

I actually have written prototypes (among others) of

  • new-document-wizards
  • spell checkers
  • importers for exotic file formats
  • connectors to databases and content management systems
  • GUI hacks (full screen mode, tear-off menus etc.)

Current problems

As you see in the current state scripter is already powerful but it lacks

  • full integration into the GUI and the core: Scripts cannot easily add menu entries or toolbar icons, save settings into the Scribus configuration file, etc. [CR: This will probably need a decently designed public C++ API for Scribus, plus a SIP wrapper for it]
  • completeness: While writing scripts you sometimes find things which are still missing. For example there is a function to zoom the document view. But there lacks a function to get the current zoom factor. [CR: Again, need a real public API in C++ to make any kind of completeness effort viable and maintainable]
  • ease of use: The script console is really minimalistic and needs improvement. Also there is no standard way to distribute and install scripts.
  • security: Scripter should warn you if a script wants to import a module which can access the file system. The user should be asked if he trusts a script. [CR: unfortunately the first part of this is not viable, though the second is - see Scripter Security ]

How to make scripting situation better

So what is needed?

For the user

  • a standard way to find and to install extensions and scripts from the web, preferably from a trusted repository [CR: Great idea, especially if combined with signing of scripts for integrity etc]
  • scripts look and behave as native parts of Scribus

For the developer

  • specifications on how scripts can be integrated with Scribus
  • more example scripts / extensions to see what is possible and how to make use of Scribus' features
  • completed missing functionally


My idea

So I propose writing an extension manager for Scribus together with some example scripts and extensions.

This will probably help to find missing functionally and integration issues. Currently it is very easy to write scripts and this should not become harder by adding this new layer.

Mozilla Firefox shows what is possible with easily installable extensions. People without knowledge of C-programming can realize crazy ideas and useful features. A lot of the popularity of Firefox comes from its add-ons.

[CR: discussion moved to The talk page for this article.]

Implementation details

The GUI part will use the PyQt library which can directly integrate with the Scribus GUI. PyQt is available on all platforms where Qt is available and also supports Qt4 which Scribus will soon use. Nearly all Linux distributions contain PyQt and some like Kubuntu install it by default.

Most parts of the extension manager can be implemented in pure Python and rely on the already available hooks in scripter. Besides some stuff has to be added:

  • functions to get the configuration directory and the program directory
  • some widgets need to have a name so PyQt can find and access them
  • Bindings for the preferences manager [CR]
  • Bindings for the menu manager [CR]
  • (more to come soon)

See the links below to see that I already have experience in this area.

Links



Excerpt from submitted abstract and application to GSoC 2007

Rationale

Scribus shows that serious desktop publishing is possible with free software. Scribus has many powerful features which are commonly mentioned in articles, e.g. the high quality PDF export and the color management. But the potential of the scripting interface is mostly underestimated. The Scripter plug-in is included in the Scribus main distribution together with some example scripts. However it does not get the attention it deserves in my point of view. Because of that I want to show what is possible with the Python scripting interface in Scribus.

Overview

Other open-source projects like Firefox, Blender, Inkscape, OpenOffice or Gimp have shown that bindings to scripting languages result in many new features and contribute largely to the success of the application. People without knowledge of C/C++ but some Python-, JavaScript- or Perl-skills can easily write import-/export-filters, create wizards, write plug-ins or realize something nobody else has thought about before. The distributed scripts can be used without recompiling and even without knowledge of the scripting language.

Scripts can help to automate tasks and can fit the special needs of a user. Also it is possible to test new ideas. Languages like Python are great for prototyping. And sometimes it makes more sense to stay with Python instead of rewriting everything in a compiled language.

As Scribus already provides a good basis with its Scripter plug-in I want to improve it and its usability. To show the power of the scripting interface I want to write some small and some larger example scripts (resp. extension scripts). My idea is that Scribus needs a script-/extension-manager like Firefox has for themes and add-ons. Like Firefox a user can browse the scripts on a website and install them by just clicking a link. So my task will also include building a small web application to provide a repository of scripts.

So the project consists of three tasks which are mutually dependent. 1. The script-manager based on PyQt and its QTextBrowser 2. The example scripts and extensions. Planed are an OpenClipArt browser and a simple spell-checker. 3. The repository. Either a CGI or a script which generates static pages

At each task I will determine which functionally is missing and how these sub-projects can be linked together. This will show what needs to be done in the scripting interface. Currently scripts do not look like native Scribus components because e.g. they cannot be connected to menu-/toolbar-entries or context-menus. There is also no standard way to install scripts to make them automatically available. These things need to be addressed. Some implementation issues will be discussed with the Scribus team, e.g. about security (already partly done), integration or future directions. Currently there are thoughts in the Scribus team about creating new automatic bindings. Because of that I will do two things: 1. I will try to keep my code modular and flexible. It should be relatively easy to port the script-/extension-manager to the new bindings when available. 2. I will evaluate different binding generators (sip, swig, boost, smoke) and make tests to see which will be the best path to go (probably sip with a separate C++ API)


Benefits

1. Firefox showed that people liked the flexibility to install add-ons. My project will bring this feature to Scribus in the hope that people will enjoy Scribus even more and it will attract new users. 2. The script interface and its experience will improve which will hopefully attract new script writers. 3. Scribus gets a repository for scripts anyone can easily access. I would be willing to host it after the GSoC. 4. Some C++ code is not needed anymore and could probably be removed, e.g. the script console. I already wrote a better replacement in PyQt.


Deliverables

  • a script manager
  • example scripts
  • a script repository
  • documentation for the script writers
Retrieved from "https://wiki.scribus.net/wiki/index.php?title=Improve_scripting_and_its_experience_in_Scribus&oldid=24040"