This chapter describes the basic requirements basic requirements for inclusion in the FreeDict Project. Before we can accept a dictionary, we ask that it meets a common set of specifications. This section gives an overview of those requirements. Essentially, a dictionary must meet clear copyright and licensing guidelines and must be presented in a particular XML format. The FreeDict project including all content, documentation and tools is Free SoftwareFree Software. Any dictionary to be considered for archiving and release by this project must first be Free, Open Source Software. The developers of a dictionary may always hold the copyright to their particular dictionary. There are a number of available licenses that can readily achieve this for you. This means you must release the dictionary under the terms and conditions of the GNU General Public License (GPL Version 2 or later), or a License of compatible type. Alternatively you may formally release the dictionary to the public domainPublic Domain. You should understand that in doing so you have no easy way of stopping anybody from capturing your content and locking it up under their own Copyright and/or License. We don't recommend this. We do recommend the GPL Version 2 or the GNU Free Documentation License. Of course it's your choice. Both of these licenses (and those that are compatible with them), protect you and your Copyright. Briefly; they say you are releasing the dictionary: 1/ In the hope that it will be useful, 2/ With no warranty to it's fitness or otherwise for any use, and 3/ You allow free use and access to the work, provided the user passes on the same right if any substantial part of the work is included in their own. Please understand that FreeDict will not agree to any license that does not meet or agree with the aims and scope of the GPL2GPL2. For current examples of licenses that do agree (and some that don't), please see: http://www.gnu.org/licenses/license-list.html. FreeDict dictionaries are marked up using the XML version of the Text Encoding Initiative DTD, Chapter 12 (Dictionaries). This may seem a little daunting at first, but please read on as we are working quickly to make this much easier. Full instructions are given in the "Writing a FreeDict Dictionary" section. We are also developing and testing tools that may be suitable for automating most of this. Text Exchange Initiative TEI DTD There are many advantages to using a standard content based approach like this: Inherits most of the advantages of XML including: content based rather than layout based application independent platform independent further processing readily possible across the entire FreeDict collection enables full use of existing or customised XML technologies Standardises input and output formats. Protects against obsolescence TEI has comprehensive DTDs available. The Dictionary DTD is just one of a very wide conceptual set. Elements already exist for lexicographic, etymological, phonetic and other particularities of dictionaries. The TEI XML combination allows processing, development and use beyond the immediate scope of the FreeDict translating dictionaries. TEI technologies are reasonably well understood and used in academic circles. Like for anything else, using TEI XML bears disadvantages: High memory requirements, for storage as well as for processing The TEI DTD is too permissive. It allows too complex content models for its elements, because it was written to capture as many existing texts as possible. Since almost all elements are allowed inside others, writing software to further process TEI data becomes complex. FreeDict uses its own subset of the TEI DTD. This subset wil be defined in this Howto, once it is stable. Till then it is described only. TEI is missing typologies. Eg. TEI does not prescribe to encode the Part of Speech of a noun as "noun" or "n" or anything else. Another missing typology is one for Cross References, eg. "synonym", "hypernym" etc. FreeDict has to define them. XML data requires more than a text editor for easy maintenance due to its verbosity. Eg. you cannot enter entries speedily when you have to enter all tags manually. To ease this, FreeDict develops the (as of yet unreleased) FreeDict-Editor. People in different countries use different characters to represent the words of their native languages. Nowadays most applications, including email systems and web browsers, are 8-bit clean, i.e. they can operate on and display text correctly provided that it is represented in an 8-bit character set, like ISO-8859-1. Bruno Haible: Linux Unicode Howto Bruno Haible Unicode From: http://www.unicode.org/standard/WhatIsUnicode.html Unicode provides a unique number for every character, no matter what the platform, no matter what the program, no matter what the language. Essentially, Unicode encoding supplies the unique character number that is typed to the page each time you enter a letter on your keyboard. A font-set then renders the character onto the screen so that you may read it. It's important to understand that Unicode is not a font-set, it is a protocol for the mappings that font-sets use to render fonts on your screen or on to a printed page. All new dictionaries should be written using the Unicode character set. If you use a text editor that is unicode capable, all should be well. UTF-8 is a way of wrapping up all real-world characters in a portable and efficient way. This includes most 8 bit and many 16 bit or 2 byte character sets. Your current character sets are probably included, so it may be as simple as putting <?xml version="1.0" encoding="UTF-8" standalone="no"?> as the XML declaration of your final document. You should ensure that your dictionary does not rely on any particular font set and is equally functional when rendered as simple text. Remember, fonts are just "pretty renderings" of real characters. Most modern Text Editors (e.g. Xemacs, emacs, Vim, GEdit, Kedit, notepad) should be fine. This is not the place for a full explanation of Unicode. Please see Markus Kuhn's excellent summary at http://www.cl.cam.ac.uk/~mgk25/unicode.htm. The Linux Unicode HOWTO is well worth visiting: ftp://ftp.ilog.fr/pub/Users/haible/utf8/Unicode-HOWTO.html encoding UTF-8 Web Browsers Unicode Compliance Universal Character Set UCS-2 UTF-8 ISO IEC ISO 10646 ISO The Universal Character Set is defined by ISO/IEC standard ISO 10646. Unicode is a parallel standard that functionally defines a few "coal face" or real life, protocols. Unicode is kept in synchronisation with ISO/IEC 10646. UTF-8 is one of four ways to save Unicode into bytes. Another common one is UTF-16. UTF-8 is the default standard for XML documents and is an excellent choice as it contains the character mappings sets from (almost) all known languages, while being fully compliant with current and earlier computer standards. Note most web browsers and protocols assume a UTF-8 compliant encoding. All ISO type character sets in general use are covered by UTF-8 (including ASCII) as the KJC Family (Korean, Japanese and some of the Chinese family of characters). If your editor has a UTF-8 option please set it on. If you have automated some or all of your dictionary construction, please be careful to maintain character type compatibility throughout the process. C coders should use the wide character type. Most scripting languages now also support UTF-8 (Python, Perl, PHP, Java and Ruby at least). Shell scripts usually adopt the local environment settings. Please check your gawk and sed are mapping cleanly. You may need very recent versions. Scripting Languages Python, Perl, Java, PHP, Ruby Awk Sed Sh ENV If you are on a Linux (or similar) system, try man 7 unicode. You may also have some unicode tools on board: man 1 unicode. You may also need to set your LANG environment settings, most Linux type systems support doing this on a per instance basis, that is you may run a number of language and locales concurrently. Examples are given later. locale LANG Linux Manual Page