Installation¶
You can download Frog, manually compile and install it from source. However, due to the many dependencies and required technical expertise this is not an easy endeavor.
Linux users should first check whether their distribution’s package manager has up-to-date packages for Frog, as this provides the easiest way of installation.
If no up-to-date package exists, we recommend to use LaMachine. Frog is part of our LaMachine software distribution and includes all necessary dependencies. It runs on Linux, BSD and Mac OS X. It can also run as a virtual machine under other operating systems, including Windows. LaMachine makes the installation of Frog straightforward; detailed instructions for the installation of LaMachine can be found here: http://proycon.github.io/LaMachine/.
Manual compilation and installation¶
The source code of Frog for manual installation can be obtained from GitHub. Because of file sizes and to cleanly separate code from data, the data and configuration files for the modules of Frog have been packaged separately.
- Source code repository: https://github.com/LanguageMachines/frog/
- Stable releases [1]: https://github.com/LanguageMachines/frog/releases/
- Frog data repository: https://github.com/LanguageMachines/frogdata/ (required dependency!)
To compile these manually, you first need current versions of the following dependencies of our software, and compile and install them in the order specified here:
ticcutils
[2] - A shared utility librarylibfolia
[3] - A library for the FoLiA formatucto
[4] - A rule-based tokenizertimbl
[5] - The memory-based classifier enginembt
[6] - The memory-based taggerfrogdata
[7] - Datafiles needed to run Frog
You will also need the following third party dependencies:
- icu - A C++ library for Unicode and Globalization support. On
Debian/Ubuntu systems, install the package
libicu-dev
. - libxml2 - An XML library. On Debian/Ubuntu systems install the
package
libxml2-dev
. - textcat - A library for language detection. On Debian/Ubuntu systems install the
package
libexttextcat-dev
. - A sane build environment with a C++ compiler (e.g. GCC or Clang), autotools, autoconf-archive, libtool, pkg-config
The actual compilation proceeds by entering the Frog directory and issuing the following commands:
$ bash bootstrap.sh
$ ./configure
$ make
$ sudo make install
/usr/local/
by default),
you may use the –prefix
option in the configure step:./configure –prefix=/desired/installation/path/
.Quick start guide¶
Frog aims to automatically enrich Dutch text with linguistic information of various forms. Frog integrates several NLP modules that perform the following tasks: tokenize text to split punctuation from word forms (including recognition of sentence boundaries and multi-word units), assignment of part-of-speech tags, lemmas, and morphological and syntactic information to words.
We give a brief explanation on running Frog to get you started quickly, followed by a more elaborate description of using Frog and how to manipulate the settings for each of the separate modules in Chapter: Frog Modules.
Frog is developed as a command line tool. We assume the reader already has at least basic command line skills.
Typing frog -h
on the command line results in a brief overview of all
available command line options. Frog is typically run on an input
document, which is specified using the -t option for plain
text, –JSONin for
documents in JSON format, or -x for documents in the FoLiA XML format. It is, however,
also possible to run it interactively or as a server. We show an example
of the output of Frog when processing the contents of a plain-text file
test.txt
, containing just the sentence In ’41 werd aan de stamkaart
een z.g. inlegvel toegevoegd.
We run Frog as follows: $ frog -t test.txt
Frog will present the output as shown in the example below:
1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
---|---|---|---|---|---|---|---|---|---|
1 | In | in | [in] | VZ(init) | 0.987660 | O | B-PP | 0 | ROOT |
2 | ’41 | ‘41 | [‘41] | TW(hoofd,vrij) | 0.719498 | O | B-NP | 1 | obj1 |
3 | werd | worden | [word] | WW(pv,verl,ev) | 0.999799 | O | B-VP | 0 | ROOT |
4 | aan | aan | [aan] | VZ(init) | 0.996734 | O | B-PP | 10 | mod |
5 | de | de | [de] | LID(bep,stan,rest) | 0.999964 | O | B-NP | 6 | det |
6 | stamkaart | stamkaart | [stam][kaart] | N(soort,ev,basis,zijd,stan) | 0.996536 | O | I-NP | 4 | obj1 |
7 | een | een | [een] | LID(onbep,stan,agr) | 0.995147 | O | B-NP | 9 | det |
8 | z.g. | z.g. | [z.g.] | ADJ(prenom,basis,met-e,stan) | 0.500000 | O | I-NP | 9 | mod |
9 | inlegvel | inlegvel | [in][leg][vel] | N(soort,ev,basis,zijd,stan) | 1.000000 | O | I-NP | 10 | obj1 |
10 | toegevoegd | toevoegen | [toe][ge][voeg][d] | WW(vd,vrij,zonder) | 0.998549 | O | B-VP | 3 | vc |
11 | . | . | [.] | LET() | 1.000000 | O | O | 10 | punct |
The ten TAB-delimited columns in the output of Frog contain the information we list below. This columned output is intended for quick interpretation on the terminal or in scripts. It does, however, not contain every detail available to Frog.
- Token number
- (Number is reset every sentence.)
- Token
- The text of the token/word
- Lemma
- The lemma
- Morphological segmentation
- A morphological segmentation in which each morpheme is enclosed in square brackets
- PoS tag
- The Part-of-Speech tag according to the CGN tagset [VanEynde2004].
- Confidence
- in the PoS tag, a number between 0 and 1, representing the probability mass assigned to the best guess tag in the tag distribution
- Named entity type
- in BIO-encoding [8]
- Base phrase chunk
- in BIO-encoding
- Token number of head word
- in dependency graph (according to the Frog parser)
- Dependency relation type of the word with head word
For full output, you will want to instruct Frog to output to a FoLiA XML file. This is done using the -X option, followed by the name of the output file.
To run Frog in this way we execute: $ frog -t test.txt -X test.xml The result is a file in FoLiA XML format [FOLIA] that contains all information in a more structured and verbose fashion. More information about this file format, including a full specification, programming libraries, and other tools, can be found on https://proycon.github.io/folia. We show an example of the XML structure for the token aangesneden in the XML example below and explain the details of this structure in the :doc: Folia Documentation<https://folia.readthedocs.io/en/latest/introduction.html#annotation-types>. Each of the layers of linguistic output will be discussed in more detail in the Chapter :doc: Frog Modules<moduleDetails>.
<w xml:id="WP3452.p.1.s.1.w.4" class="WORD">
<t>aangesneden</t>
<pos class="WW(vd,vrij,zonder)" confidence="0.875" head="WW">
<feat class="vd" subset="wvorm"/>
<feat class="vrij" subset="positie"/>
<feat class="zonder" subset="buiging"/>
</pos>
<lemma class="aansnijden"/>
<morphology>
<morpheme>
<t>aan</t>
</morpheme>
<morpheme>
<t>ge</t>
</morpheme>
<morpheme>
<t>snijd</t>
</morpheme>
<morpheme>
<t>en</t>
</morpheme>
</morphology>
</w>
By default the output of Frog is written to screen (i.e. standard output). There are two options for writing the output to file:
-o <filename>
– Writes columned (TAB delimited) data to file.-X <filename>
– Writes FoLiA XML to file.
We already saw the input option -t <filename>
for plain-text files. It
is also possible to read FoLiA XML documents instead, using the `` -x
<filename>`` option. Frog also allows for inputfiles and outputfiles in
JSON format, when using the options --JSONin
or
--JSONout
respectively. We show an example of JSON input and output
below. Each linguistic layer in the output is presented as a key-value pair for each
detected token in the text input.
JSON input: [{“sentence”:”Dit nog zo’n boeiende test.”}]
JSON output: [
- {
- “chunking”: {
- “confidence”: 1.0, “tag”: “B-NP”
}, “index”: 1, “lemma”: “dit”, “morph”: “[dit]”, “parse”: {
“parse_index”: 2, “parse_role”: “su”}, “pos”: {
“confidence”: 0.7770847770847771, “tag”: “VNW(aanw,pron,stan,vol,3o,ev)”}, “ucto”: {
“new_paragraph”: true, “token”: “WORD”}, “word”: “Dit”
}, {
- “chunking”: {
- “confidence”: 1.0, “tag”: “B-VP”
}, “index”: 2, “lemma”: “zijn”, “morph”: “[zijn] “parse”: {
“parse_index”: 0, “parse_role”: “ROOT”}, “pos”: {
“confidence”: 0.9998905788379473, “tag”: “WW(pv,tgw,ev)”}, “ucto”: {
“token”: “WORD”}, “word”: “is”
}, {
- “chunking”: {
- “confidence”: 1.0, “tag”: “B-NP”
}, “index”: 3, “lemma”: “een”, “morph”: “[een]”, “parse”: {
“parse_index”: 4, “parse_role”: “det”}, “pos”: {
“confidence”: 0.9991126885536823, “tag”: “LID(onbep,stan,agr)”}, “ucto”: {
“token”: “WORD”}, “word”: “een”
}, {
- “chunking”: {
- “confidence”: 1.0, “tag”: “I-NP”
}, “index”: 4, “lemma”: “test”, “morph”: “[test]”, “parse”: {
“parse_index”: 2, “parse_role”: “predc”}, “pos”: {
“confidence”: 0.9030552291421856, “tag”: “N(soort,ev,basis,zijd,stan)”}, “ucto”: {
“space”: false, “token”: “WORD”}, “word”: “test”
}, }, {
- “chunking”: {
- “confidence”: 1.0, “tag”: “O”
}, “index”: 5, “lemma”: “.”, “morph”: “[.]”, “parse”: {
“parse_index”: 4, “parse_role”: “punct”}, “pos”: {
“confidence”: 1.0, “tag”: “LET()”}, “ucto”: {
“token”: “PUNCTUATION”}, “word”: “.”
}
]
Besides input of a single plain text file, Frog also accepts a directory of plain text files (or JSON format) as input –testdir=<directory> , which can also be written to an output directory with parameter –outputdir=<dir>. The FoLiA equivalent for –outputdir is –xmldir. To read multiple FoLiA documents, instead of plain-text documents, from a directory, use -x –testdir=<directory>.
Frog can be started in an interactive mode by simply typing frog
on
the command line. Frog will present a frog>
prompt after which you
can type text for processing. By default, you will press ENTER at an
empty prompt before Frog will process the prior input. This allows for
multiline sentences to be entered. To change this behavior, you may want
to start Frog with the -n option instead, which tells it to assume each
input line is a sentence. FoLiA input or output is not supported in
interactive mode.
To exit this mode, type CTRL-D.
Frog offers a server mode that launches it as a daemon to which multiple
clients can connect over TCP. The server mode is started using the
-S <port>
option. Note that options like -n
and –skip
are
valid in this mode too.
You can for example start a Frog server on port 12345 as follows:
$ frog -S 12345
.
The simple protocol clients should adhere to is as follows:
- The client sends text to process (may contain newlines)
- The client sends the string
EOT
followed by a newline - The server responds with columned, TAB delimited output, one token per line, and an empty line between sentences.
- FoLiA input and output are also possible, using the
-x
and-X
options without parameters. When-X
is selected, TAB delimited output is suppressed. - The last line of the server response consists of the string
READY
, so the client knows it received the full response.
Communicating with Frog on such a low-level may not be necessary, as there are already some libraries available to communicate with Frog for several programming languages:
- Python – pynlpl.clients.frogclient [9]
- R – frogr [10] – by Wouter van Atteveldt
- Go – gorf [11] – by Machiel Molenaar
The following example shows how to communicate with the Frog server from
Python using the Frog client in PyNLPl, which can generally be installed
with a simple pip install pynlpl
, or is already available if you use
our LaMachine distribution.
from pynlpl.clients.frogclient import FrogClient
port = 12345
frogclient = FrogClient('localhost',port)
for data in frogclient.process("Dit is de tekst om te verwerken.")
word, lemma, morph, pos = data[:4]
#TODO: Further processing per word
Do note that Python users may prefer using the python-frog
binding
instead, which will be described in the Chapter Python Frog. This binds
with Frog natively without using a client/server model and therefore has
better performance.
[1] | The source code repository points to the latest development version by default, which may contain experimental features. Stable releases are deliberate snapshots of the source code. It is recommended to grab the latest stable release. |
[2] | https://github.com/LanguageMachines/ticcutils |
[3] | https://github.com/LanguageMachines/libfolia |
[4] | https://languagemachines.github.io/ucto |
[5] | https://languagemachines.github.io/timbl |
[6] | https://languagemachines.github.io/mbt |
[7] | https://github.com/LanguageMachines/frogdata |
[8] | B (begin) indicates the begin of the named entity, I (inside) indicates the continuation of a named entity, and O (outside) indicates that something is not a named entity |
[9] | https://github.com/proycon/pynlpl, supports both Python 2 and Python 3 |
[10] | https://github.com/vanatteveldt/frogr/ |
[11] | https://github.com/Machiel/gorf |