Building Android code search server based on opengrok

table of Contents

1. Introduction to OpenGrok

2. Install OpenGrok

   2.1 Install JAVA runtime environment

   2.2 Install Web Server-Tomcat

   2.3 Install OpenGrok

   2.4 Configure OpenGrok

   2.5 Install universal-ctags

   2.6 Build Source Index

   2.6 Update source index

1. Introduction to OpenGrok

OpenGrok is a fast and usable source code search and cross reference engine. It helps you search, cross-reference and navigate your source tree

Requirements:

• Latest Java 1.8
• A servlet container like GlassFish or Tomcat (8.x or later) also running with Java at least 1.8
• Universal ctags (Exuberant ctags work too)

OpenGrok is a fast and easy-to-use source search engine and comparison engine.It can help us quickly search, locate and compare code trees.

system requirement:

• Java 1.8
• Web server GlassFish or Tomcat (8.x or later, here Tomcat8)
• Universal ctags (Exuberant ctags also work, there are pits here, in fact Exuberant ctags do not work, the latest version of OpenGrok will detect whether there are Universal ctags when indexing, and no direct error is reported)

In short, OpenGrok is a millisecond-level code search tool, and its biggest advantage is that the search speed is fast.

2. Install OpenGrok

2.1 Install JAVA Runtime Environment

OpenGrok and Tomcat both rely on JAVA, so we first need a JDK to support its operation

Code

sudo apt-get update
sudo apt-get install openjdk-8-jdk
java -version // Check if java is installed correctly

For specific installation, please refer to the detailed installation of Ubuntu JDK

2.2 Install Web Server-Tomcat

OpenGrok is web-based access and requires a web system to support it, so a web server must be installed.

The Tomcat8 package is already provided in the source of Ubuntu18.04, and Tomcat8 is installed directly from the source.

Code

sudo apt-get install tomcat8
Start Tomcat8

sudo service tomcat8 start // Start the Tomcat service
or

sudo /etc/init.d/tomcat8 start
After starting the Tomcat service, enter the URL in the browser and see that the page displays "It works!"

http://localhost: 8080

Tomcat8 common commands

Code

sudo service tomcat8 start // Start the Tomcat service
sudo service tomcat8 stop // Close the Tomcat service
sudo service tomcat8 status // View Tomcat service status
sudo service tomcat8 restart // Restart the Tomcat service

For more detailed installation, please refer to Ubuntu installation and configuration tomcat8 service

2.3 Install OpenGrok

After installing Tomcat, the next step is to configure OpenGrok.

OpenGrok Download Address:

http://opengrok.github.io/OpenGrok

https://github.com/oracle/opengrok/releases

You can download the compiled file of OpenGrok, or you can download the source file at this URL. Here we directly download the compiled file (the students who are interested in OpenGrok can download the source code to see what it is). After downloading, decompress it with the following command:

tar xvzf opengrok-1.1-rc41.tar.gz -C/opt

2.4 Configure OpenGrok

If you want OpenGrok to run properly, you need a lot of environment variables. If we install jdk, Tomcat and OpenGrok according to the requirements of OpenGrok, and establish a directory structure, these environment variables will be set correctly when running the OpenGrok script, but if we If you want more flexible configuration, you still have to customize the directory structure, and manually configure these environment variables.

The environment variables are configured as follows: opengrokSetEnv.sh

Code

# ------------------------------------------------- ------------------------------
#-JAVA_HOME Full Path to Java Installation Root
#-JAVA Full Path to java binary (to enable 64bit JDK)
#-JAVA_OPTS Java options (e.g. for JVM memory increase
# ------------------------------------------------- ------------------------------
JAVA_HOME =/usr/lib/jvm/java-8-openjdk-amd64
JRE_HOME = $ JAVA_HOME/jre
JAVA_BIN = $ JAVA_HOME/bin
CLASSPATH = $ JAVA_HOME/lib/dt.jar: $ JAVA_HOME/lib/tools.jar: $ JRE_HOME/lib
PATH = $ PATH: $ JAVA_HOME/bin: $ JRE_HOME/bin
export JAVA_HOME JRE_HOME PATH CLASSPATH

# ------------------------------------------------- ------------------------------
#-OPENGROK_DISTRIBUTION_BASE Base Directory of the OpenGrok Distribution
#-OPENGROK_INSTANCE_BASE Base Directory of the OpenGrok User Data Area
#-EXUBERANT_CTAGS Full Path to Exuberant CTags
#-OPENGROK_CTAGS_OPTIONS_FILE Full path to file with extra command line
# options for CTags program (for its --options
# ------------------------------------------------- ------------------------------
# opengrok home directory
export OPENGROK_INSTANCE_BASE =/opt/opengrok
export SCRIPT_DIRECTORY = $ OPENGROK_INSTANCE_BASE/bin
export OPENGROK_DISTRIBUTION_BASE = $ OPENGROK_INSTANCE_BASE/lib


# source code root
export SRC_ROOT = $ OPENGROK_INSTANCE_BASE/database/src
# generated data root
export DATA_ROOT = $ OPENGROK_INSTANCE_BASE/database/data
#
EXUB_CTAGS =/usr/bin/ctags


# ------------------------------------------------- ------------------------------
#-OPENGROK_APP_SERVER Application Server ("Tomcat" or "Glassfish")
#-OPENGROK_WAR_TARGET_TOMCAT Tomcat Specific WAR Target Directory
#-OPENGROK_WAR_TARGET_GLASSFISH Glassfish Specific WAR Target Directory
#-OPENGROK_WAR_TARGET Fallback WAR Target Directory
#-OPENGROK_TOMCAT_BASE Base Directory for Tomcat (contains webapps)
#-OPENGROK_GLASSFISH_BASE Base Directory for Glassfish
# (contains domains)
# ------------------------------------------------- ------------------------------
export OPENGROK_APP_SERVER = Tomcat
export OPENGROK_TOMCAT_BASE =/var/lib/tomcat8
export OPENGROK_WAR_TARGET_TOMCAT = $ OPENGROK_TOMCAT_BASE/webapps
export OPENGROK_WAR_TARGET = $ OPENGROK_TOMCAT_BASE/webapps
export CATALINA_HOME = $ OPENGROK_TOMCAT_BASE

Once completed, you can use the opengrokSetEnv.sh script to set environment variables every time before running OpenGrok

Code

source opengrokSetEnv.sh

For information on OpenGrok scripts, see https://github.com/OpenGrok/OpenGrok/blob/master/OpenGrok

If the error occurs, please check the source code to set the environment variable information.

After decompression, go to the decompression directory, and copy the source.war package in the/opt/opengrok/lib directory to the apache-tomcat/webapps directory

Enter in browser

http://localhost:8080/source/

If you see the OpenGrok page, it proves that OpenGrok is running successfully.

Next you need to change the web.xml configuration file in the apache-tomcat/webapps/source/WEB-INF directory

Code

<context-param>
   <param-name> CONFIGURATION </ param-name>
   <param-value> /opt/opengrok/etc/configuration.xml </ param-value>
   <description> Full path to the configuration file where OpenGrok can read it's configuration </ description>
 </ context-param>

Configure the path of configuration.xml to the/opt/opengrok/etc/configuration directory.This configuration.xm will be automatically generated in the following source code indexing steps.

2.5 Install universal-ctags

Code

sudo apt install autoconf
cd/tmp
git clone https://github.com/universal-ctags/ctags
cd ctags
./autogen.sh
./configure --prefix =/opt/universal-ctags # My installation path. You adjust to your situation.
make -j8
sudo make install
Update the ctags executable to the system PATH? No, I choose how to create the link:

# If you have Exuberant ctags installed, first delete the linked files:
# sudo mv /usr/bin/ctags /usr/bin/ctags.bak

# Then, link over the newly compiled and installed universal-ctags:
sudo ln -s /opt/universal-ctags/bin/ctags/usr/bin/ctags

2.6 Build Source Index

Next, we need to configure the index for our source code. OpenGrok generates the index information of the source code. It looks like it is to establish a relevant database to achieve the purpose of fast search.

Set the following environment variables

Environment variable Description Default value
SRC_ROOT The source code path of the index to be generated $ {OPENGROK_INSTANCE_BASE}/src
DATA_ROOT Path to the generated index $ {OPENGROK_INSTANCE_BASE}/data
Then we can directly create the src and data directories in the opengrok installation directory/opt/opengrok

Then we put the source code that needs to be indexed in the src directory. Of course, we don’t really need to put the source code here, we just need to create a link for it.

Code

cd /opt/opengrok/src
ln -s /usr/src/android8.0 android8.0

Generate index

Code

indexer.py -J = -Djava.util.logging.config.file =/opt/opengrok/logging.properties \
    -a /opt/opengrok/lib/opengrok.jar-\
    -s/opt/opengrok/src -d/opt/opengrok/data -H -P -S -G \
    -W /opt/opengrok/etc/configuration.xml -U http: // localhost: 8080

Then we open

http://localhost:8080/source

You can see our source code

2.6 Update source index

When the code changes, the source code index needs to be updated to reflect the changes to Opengrok

Code

touch/opt/opengrok/src/*

cd/opt/opengrok/bin

indexer.py -J = -Djava.util.logging.config.file =/opt/opengrok/logging.properties \
    -a /opt/opengrok/lib/opengrok.jar-\
    -s/opt/opengrok/src -d/opt/opengrok/data -H -P -S -G \
    -W /opt/opengrok/etc/configuration.xml -U http: // localhost: 8080