Manual:Manual for administrators

From Urantiapedia

About Urantiapedia

Urantiapedia is a collaborative website based on MediaWiki with the purpose of serving as a unified center for the dissemination of all knowledge related to The Urantia Book.

About Blue Fields

Blue Fields is a website that has been created to unite and organize readers of The Urantia Book who wish to participate in intelligent, purposeful and altruistic projects in all the world.

About GitHub

The Urantiapedia project on GitHub (https://github.com/JanHerca/urantiapedia) is a project created to serve as a quick start-up procedure for the Urantiapedia. It contains files in certain formats and software that help automate the creation of initial content.

Who is this manual for?

This manual is intended for users called administrators, that is, those users who meet the following requirements:

  • are registered users with an account at Blue Fields and at GitHub.
  • are users who have been authorized as project administrators by the team that runs the project on the platform Blue Fields.
  • are registered users with an account on the page Urantiapedia. During the project preparation phase this registration can only be done by other existing administrator users with an account on Urantiapedia.
  • They are users with extensive knowledge of GitHub and MediaWiki.
  • are users with extensive knowledge of The Urantia Book in at least one language in which it is published, and preferably with a good level of the English version.
  • wish to cooperate in a selfless way in the incorporation of information related to The Urantia Book in the Urantiapedia.

The website administrators are the only ones in charge of uploading changes to the Urantiapedia website to check the results of the work during the preparation phase.

Workflow

About the phases of the project, the working procedure, the content planned for the project and how it is organized, read the Manual:Manual for GitHub editors.

The list of tasks to be carried out is as follows:

  1. Translation of Paramony from English to the target language.
  2. Translation and revision of the Topic Index from English to the target language.
  3. Obtaining a translation of The Bible in the target language.
  4. Converting The Urantia Book from HTML to JSON without footnotes.
  5. Adding footnotes to The Urantia Book in JSON.
  6. Conversion of the Topic Index from TXT to Wikitext.
  7. Conversion of The Urantia Book from JSON with footnotes to Wikitext including references to topics.
  8. Converting The Bible from LaTeX to Wikitext with footnotes.

The GitHub editors will perform tasks 1-3. The administrators will handle tasks 4-8. We detail each administrators task in the following sections.

Converting The Urantia Book from HTML to JSON without footnotes

The input files in HTML format must be in the input/html/book-xx folders, where xx are the two letters of the language (es = Spanish, en = English, etc.)

IMPORTANT: The HTML files downloaded from the official website of the Urantia Foundation may have some problems to be processed as they are and it is necessary to introduce small modifications to help in the process of converting them to the JSON format.

Once the files have been downloaded from the Urantia Foundation website, there should be one file (*.html) for each The Urantia Book paper, a total of 197 files. The name of each file with each paper must meet the pattern that the last three characters of the name are the paper number, from 000 to 196 for the conversion to work later. For example, in French the file with paper 1 is called 06-Fascicule001.html; the same file in Bulgarian is called 06-Документ001.html.

Since in HTML format The Urantia Book can adopt different HTML tags for what is considered the title of the paper, the title of a section, etc., in the program that performs the conversion it may be necessary to include some modifications when a new language is added. This work will be carried out by the developer users, who are the users who are in charge of the development of the Urantiapedia Tools. This software is developed in Javascript and the code is in the app folder of the project.

Files for 24 languages ​​are currently available: Bulgarian (book-bg), Czech (book-cs), Danish (book-da) , German (book-de), Greek (book-el), English (book-en), Spanish (book-es, book-es-1993 and shortly book-es-2021), Estonian (book-et), Farsi (book-fa), Finnish (book-fi), French (book-fr), Hebrew (book-he), Hungarian (book-hu), Indonesian (book-id), Italian (book-it), Japanese (book-ja), Korean (book-ko), Lithuanian (book-lt), Dutch (book-nl), Polish (book-pl), Portuguese (book-pt), Romanian (book-ro), Russian (book-ru) and Swedish (book-sv). These files have already been modified and processed so that the JSON conversion program works correctly with them, and the resulting JSON files are also available.

The Farsi translation is currently in process and not all papers are available.

It should be borne in mind that there are currently three translations in the Spanish language. All three translations are going to be available on the Urantiapedia. For that reason there will be these folders:

  • book-es, with the so-called European translation, made in 2009;
  • book-es-1993, with the first Spanish translation made in 1993;
  • book-es-2021, with the new translation made in 2021.


To convert files from HTML to JSON without footnotes, follow these steps:

  • Open the Urantiapedia Tools.
  • In Process select: convert Urantia Book (HTML) to JSON.
  • Select as input folder (HTML files) one of input/html/, such as input/html/book-fr for French.
  • Select as output folder (JSON files) one with the same name but in input/json, for example input/json/book-fr for French.
  • Click on Run.
  • If everything is correct, Conversion performed successfully will appear. In case of errors, the list of errors will appear. In this case, it is advisable to notify one of the developer users in case there is a problem with the software.


Upt html json.png

The resulting files in JSON format should be saved in input/json/book-xx folders, where xx are the two letters of the language (es = Spanish, en = English, etc.)

IMPORTANT: These files should not be generated or edited by users manually. They should always be created by conversion from the HTML format.

There will be one file for each paper of The Urantia Book, a total of 197 files. The files are named DocXXX.json, where XXX is the paper number from 000 to 196.

The format of these files is JSON (https://es.wikipedia.org/wiki/JSON). There is a converter in the Urantiapedia Tools that then converts this JSON format to the final Wikitext format.

The structure of the JSON files is this:

{
    "paper_index": "<paper number, from 0 to 196>",
    "sections": [
        {
            "section_index": "<section number, from 0 if exist to N>",
            "section_ref": "<text with section ref., e.g. “0:0”>",
            "section_title": "<title of section for no-zero sections>",
            "pars": [
                {
                    "par_ref": "<Paragraph reference, e.g. “0:0.1”>",
                    "par_pageref": "<Paragraph reference by page, e.g. “1.1”>",
                    "par_content": "<Paragraph content. See below about format>"
                },
                {

                }
            ]
        },
        {

        }
    ],
    "footnotes": [
        "<array of footnotes with Bible references>"
    ],
    "paper_title": "<paper title, without saying Paper X, only the title>"
}

In the JSON files resulting from this task, the footnotes section will be empty. To complete it, you must execute the following task.

To format the text of the paragraph with some strange characters that appear in The Urantia Book in the JSON files these equivalents are used:

Character type How they are written - Italic * * (between asterisks) - Small caps $ $ (between dollar signs) - Underline (for languages ​​that do not include italic option) | (between pipes)


Adding footnotes to The Urantia Book in JSON

Adding footnotes to The Urantia Book requires that you have previously completed the following tasks:

  • Translation of Paramony from English to the target language. The result of this task is a file for each language (input/json/footnotes-book-xx.json).
  • Converting The Urantia Book from HTML to JSON without footnotes. The result of this task is a set of 197 files in a folder for each language (input/json/book-xx).

To add footnotes to JSON files, follow these steps:

  • Open the Urantiapedia Tools.
  • In Process select: Update Urantia Book Bible Refs (JSON).
  • Select as input folder (JSON files) one of input/json/, such as input/json/book-fr for French.
  • Click on Run.
  • If everything is correct, Conversion performed successfully will appear. In case of errors, the list of errors will appear. In this case, it is advisable to notify one of the developer users in case there is a problem with the software.


Upt json json.png

The output files are different from the input files to avoid that if there is any problem they are unusable. The output is saved in folders input/json/book-xx-footnotes, where xx is the identical language code used as the input folder.


Converting Topic Index from TXT to Wikitext

The Topic Index files in TXT format are created and reviewed by the GitHub editors. These files should be left in folders like input/txt/topic-index-xx where xx is the code of the target language.

To convert these TXT files to Wikitext format, follow these steps:

  • Open the Urantiapedia Tools.
  • In Process select: Convert Topic Index (TXT) to Wiki.
  • Select as input folder (TXT files) one of input/txt/topic-index-xx.
  • Select as output folder (Wiki files) one of input/wiki/topic-index-xx. If there are already Wikitext files in the output folder, the existing ones will be overwritten.
  • In Categories of topcis select ALL to process all the topics of all the categories, or alternatively, select only one category if you want to generate only the topics that belong to that category.
  • Click on Run.
  • If everything is correct, Conversion performed successfully will appear. In case of errors, the list of errors will appear. In this case, it is advisable to notify one of the developer users in case there is a problem with the software.


Upt txt wiki.png

The output files have a Wikitext format and a *.wiki extension. A file will be generated for each topic, each one inside a folder with the same name (_, A, B, etc.) as they have the input *.txt files.

The output files are already in the format that allows uploading them directly to the Urantiapedia using the importTextFiles.php command.


Conversion of The Urantia Book from JSON with footnotes to Wikitext including references to the topics

The files of The Urantia Book in JSON format with the footnotes are generated with the tool Update Bible Refs in Urantia Book (JSON), as we have already seen, and the resulting files are saved in folders input/json/book-xx-footnotes.

To convert these JSON files to Wikitext format, follow these steps:

  • Open the Urantiapedia Tools.
  • In Process select: Convert Urantia Book (JSON) + TopicIndex (TXT) to Wiki.
  • Select as input folder (TXT files) one of input/txt/topic-index-xx.
  • Select as input folder (JSON files) one of input/json/book-xx-footnotes.
  • Select as output folder (Wiki Files) one of input/wiki/book-xx. If there are already Wikitext files in the output folder, the existing ones will be overwritten.
  • In Categories of topics select ALL so that when carrying out the conversion process all the topics of all categories are used and links in The Urantia Book are added to all the topics, or alternatively, select only one category if you want to generate only links to the topics that belong to that category.
  • Click on Run.
  • If everything is correct, Conversion performed successfully will appear. In case of errors, the list of errors will appear. In this case, it is advisable to notify one of the developer users in case there is a problem with the software.


Upt json wiki.png

The output files have a Wikitext format and a *.wiki extension. 197 files will be generated, one for each The Urantia Book paper.

The output files are already in the format that allows uploading them directly to the Urantiapedia using the importTextFiles.php command.


Converting The Bible from LaTeX to Wikitext with footnotes

The The Bible files in LaTeX format must have been created in folders such as input/text/bible-xx where xx is the code of the target language.

To convert these LaTeX files to Wikitext format, follow these steps:

  • Open the Urantiapedia Tools.
  • In Process select: Convert Bible (LaTeX) + Refs (TXT) to Wiki.
  • Select as input folder (TXT files) one of input/txt/bible-refs-es.
  • Select as input folder (LaTeX files) one of input/tex/bible-xx.
  • Select as output folder (Wiki files) one of input/wiki/bible-xx. If there are already Wikitext files in the output folder, the existing ones will be overwritten.
  • Click on Run.
  • If everything is correct, Conversion performed successfully will appear. In case of errors, the list of errors will appear. In this case, it is advisable to notify one of the developer users in case there is a problem with the software.


Upt tex wiki.png

The output files have a Wikitext format and a *.wiki extension. A file will be generated for each chapter of the Bible, and the file names will follow the pattern bookname_n.wiki, where bookname is the name of the Bible book and n the chapter number. For example, a file Obadiah_1.wiki, Obadiah_2.wiki, etc. is generated.

The output files are already in the format that allows uploading them directly to the Urantiapedia using the importTextFiles.php command.


Bulk uploads with maintenance script importTextFiles.php

To upload a large number of Wikitext files to the Urantiapedia the recommended method is to use the command importTextFiles.php from MediaWiki.

To run this command in PuTTy:

cd public_html/maintenance
php importTextFiles.php -u <user> -s "Adding references" --rc --overwrite files_to_import/*.wiki

The above command updates the pages whose names match the file names of the wiki files that are in the files_to_import folder within the maintenance folder. Instead of "Adding references", put the message you want to give to all updates. In <user> put the username of the GitHub editor that has been responsible for the changes that are uploaded. In this way, the appropriate users will appear as authors in the history of each page.

More information about the command here: https://www.mediawiki.org/wiki/Manual:ImportTextFiles.php


Appendix DEPRECATED: Mass loading of pages with the DataTransfer extension

It is preferable to use the importTextFiles.php command method. It is faster and more reliable and the input format is simply wiki format.

With this other method, to import content you would have to follow these steps:

  1. Download the DataTransfer extension: https://www.mediawiki.org/wiki/Extension:Data_Transfer
  2. Copy the compressed .gz file of the extension to the extensions folder.
  3. Unzip the file.
  4. Modify LocalSettings.php by adding this line: wfLoadExtension('DataTransfer');
  5. Add the following to allow the content of the Free_Text to not be parsed: $wgDataTransferViewXMLParseFreeText = false;
  6. See an example of what should be loaded by downloading from

?Title=Special:ViewXML&titles=page_name

  1. Load using Special:ImportXML

The load is not immediate. It uses the MediaWiki job system, which means that every few pages are visited, a number of jobs are launched.

To browse the list of jobs: https://urantiapedia.site/api.php?action=query&meta=siteinfo&siprop=statistics

Change the number of jobs that run after each visit modify the LocalSettings.php adding: $wgJobRunRate = 100;

This brings the number of jobs after each visit to 100, which is a lot when the web is working with many concurrent users. It is convenient to leave it at 1 or even 0.1 when the page is already published and being massively accessed.


Appendix DEPRECATED: Converting The Urantia Book to LaTeX format

LaTeX format The Urantia Book files are no longer needed in the project, although they are described here for clarity. These files were initially the preferred format to save The Urantia Book in the project, but they have been replaced by the use of JSON files, which are easier to process using software written in Javascript, as is the case with the Urantiapedia Tools.

The files with this format, if they exist, are in the input/tex/book-xx folders, where xx are the two letters of the language (es = Spanish, en = English, etc.)

IMPORTANT: These files should not be generated or edited by users. It was created by conversion of the HTML format.

In the folders there should be one file for each paper of The Urantia Book, a total of 197 files. The files are named DocXXX.tex, where XXX is the paper number from 000 to 196.

The format of these files is LaTeX (https://es.wikipedia.org/wiki/LaTeX). Although previously both this format and the JSON format were valid formats for The Urantia Book, currently this LaTeX format is no longer necessary and it is preferable to use the JSON format, which is more manageable, organized and easy to understand, as well as less prone to making mistakes when editing, especially if you use software that supports checking JSON files (for example, VS Code).

The structure of the files is this:

\chapter{<paper title, without saying Paper X>}
\par
%\textsuperscript{(<Paragraph reference by page, e.g. “1.1>)}
\textsuperscript{<Paragraph reference, e.g. “0:0.1”>} <Paragraph content. See below about format>
...

\section*{<section title>}
\par
%\textsuperscript{(<Paragraph reference by page, e.g. “1.1>)}
\textsuperscript{<Paragraph reference, e.g. “0:0.1”>} <Paragraph content. See below about format>
...

As you can see, this format is very simple because it simply uses the \chapter, \section and \par tags to mark the beginning of paper, section and paragraph respectively.

To include footnotes (references to The Bible) it is indicated within the content of a paragraph, as follows:

\textsuperscript{<Paragrpah ref.>} <text>\footnote{<footnote content>}<text>

E.g.:

\textsuperscript{0:2.15} 4. \textit{God the Supreme---}\footnote{\textit{God the Supreme}: Sal 136:2-3; Dn 2:47; 10:17; Jos 22:22; 1 P 2:13.}the actualizing or evolving God of time and space.

For formatting the text of the paragraph with some strange characters that appear in The Urantia Book the LaTeX equivalents that can be used are these:

Character type Way of writing - Italic \textit{<text>} - ü \"u - - --- - ‘ ’ ` ' - Line break \bigbreak - º {\textdegree} - º {\textordmasculine} - ª {\textordfeminine} - superscript (raised to 27) \textsuperscript\{27\} - superscript (cubed) \textsuperscript\{3\} - a quarter {\\textonequarter}

Links of interest

Urantiapedia user manual:

MediaWiki software: