How to create a file.
How to write data to a file.
How to create a CSV file.
How to query the following databases:
\nMySQL
PostgreSQL
MongoDB
ElasticSearch
File permission
Octal
SQL injections
Prepared queries
Schemaless
RDBMS
This section will go through different techniques to store data with Go.
\n\nThe os.Create method will create a new file :
// data-storage/file-save/simple/main.go \npackage main\n\nimport (\n "fmt"\n "os"\n)\n\nfunc main() {\n f, err := os.Create("test.csv")\n if err != nil {\n fmt.Println(err)\n return\n }\n // ... success\n}We create a file named \"test.csv\". If there is an error, we print the error and return. This program is equivalent to :
f, err := os.OpenFile("test.csv", os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)\nif err != nil {\n fmt.Println(err)\n return\n}os.Create is internally calling os.Openfile with the following parameters :
The name of the file (here: \"test.csv\")
A set of flags (here : os.O_RDWR|os.O_CREATE|os.O_TRUNC )
A file mode (here : 0666)
We will see in the next two sections what are those flags and modes. You can skip those sections if you are already familiar with those notions.
\n\nWhen you open a file with os.Openfile you are doing a system call. This system call needs to know the file path and additional information about your intentions. Those pieces of information are resumed in a list of flags. They are different types of flags :
The access mode flags : they will define in which way we will create the file. You must always give one of this flag.
The creation flags : they will affect how we will execute the system operation
The file status flags : they will have effects on all the operations that we will perform after the file opening.
| Flag type | \nFlag Name | \nDescription | \n
|---|---|---|
| \n | os.O_RDONLY | \nOpen in a read-only mode. You can just read the file | \n
| \n | os.O_WRONLY | \nOpen in write-only mode. You can write only. | \n
| \n | os.O_RDWR | \nOpen in read and write mode | \n
| \n | os.O_CREATE | \nIt will create the file if it does not exist | \n
| \n | os.O_EXCL | \nIf used with os.O_CREATE it will result in an error if the file already exists | \n
| \n | os.O_TRUNC | \nIf the file may be truncated when opened. If something is written on the file, it will be erased. | \n
| \n | os.O_APPEND | \nIf you write to the file, what you write will be appended. | \n
| \n | os.O_SYNC | \nOpen the file in synchronous mode. The system will wait that all the data has been written. | \n
Each flag is an integer. To use those flags, you have to use the bitwise operators (see section [sec:Bitmasks]). When you call os.Create the file will be opened with the flags :
os.O_RDWR|os.O_CREATE|os.O_TRUNCThis means that the file will be opened in read and write mode. It will be created if it does not exists. In addition to that, the system will truncate it if possible.
\nFor instance, to be sure that a new file has been created, you can add the flag : os.O_EXCL
f, err := os.OpenFile("test.csv", os.O_RDWR|os.O_CREATE|os.O_TRUNC|os.O_EXCL, 0666)\nif err != nil {\n fmt.Println(err)\n return\n}\nfmt.Println(f)The execution of the previous program will output :
\nopen test.csv: file existsWhich is the expected behavior
\n\nOn UNIX systems, each file has a set of permissions :
\nThe permissions attributed to the owner of the file
The permissions for users member of the group to which the file belongs
The permissions for all the other users
If you want to visualize the permissions of a file. Open your terminal and type ls -l.
\n$ ls -al\n-rw-r--r-- 1 maximilienandile staff 242 25 nov 19:47 main.go-rw-r--r-- : the file mode
1 : the number of links
maximilienandile : the owner of the file
staff : the group of the file
242 : the number of bytes of the file
25 nov 19:47 : the last modified date.
The file mode is structured in three blocks (permissions for the users, the group, and the rest of the world).
\nAs you see in figure 1 in those three blocks, you have three letters that define the permissions. They are always in the same order :
\nr : read
w : write
x : execute
The first character is a dash; if it’s a file, it’s equal to d in the case of a directory. If one of the letters is absent and only a dash is written, it means that the permission is not available.
\nLet’s take the example of the following file mode : -rw-r--r--
\nFirst char: it’s a file
User (rw-) : can read, can write, cannot execute
Group (r--) : can read, cannot write nor execute
Rest of the world (r-- ): can read, cannot write nor execute
Let’s get the file mode with go :
\n// data-storage/file-save/mode/main.go \n\n// open the file\nf, err := os.Open("myFile.test")\nif err != nil {\n fmt.Println(err)\n return\n}\n// read the file info\ninfo, err := f.Stat()\nif err != nil {\n fmt.Println(err)\n return\n}\nfmt.Println(info.Mode())The previous program will output -rw-r--r--.
\n\nThe mode can also be converted into an octal number1 (with the %o formatter) :
fmt.Printf("Mode Numeric : %o\\n", info.Mode())\nfmt.Printf("Mode Symbolic : %s\\n", info.Mode())\n// Mode Numeric : 644\n// Mode Symbolic : -rw-r--r--The numeric notation is used by Go when you call os.Openfile (third parameter). This notation is composed of 3 digits.
The digits (octal) represent a set of permissions (see figure 2). Each octal digit (from 0 to 7) has a specific signification.
\n| Permission | \nSymbolic | \nOctal | \n
|---|---|---|
| None | \n--- | \n0 | \n
| Execute only | \n--x | \n1 | \n
| Write Only | \n-w- | \n2 | \n
| Write & execute | \n-wx | \n3 | \n
| Read Only | \nr-- | \n4 | \n
| Read & execute | \nr-x | \n5 | \n
| Read & write | \nrw- | \n6 | \n
| Read & Write & Execute | \nrwx | \n7 | \n
Let’s take an example : 644. Let’s decompose it :
\nOwner : 6 which means: read + write
Group : 4 which means: read
Others : 4 which means: read
Another example 777 :
\nOwner : 7 which means: read + write + execute
Group : 7 which means: read + write + execute
Others : 7 which means: read + write + execute
On a file, you can use the method Chmod to change the file mode :
// data-storage/file-save/mode/main.go \n\nerr = f.Chmod(0777)\nif err != nil {\n fmt.Println(err)\n}Here we have changed the mode of the file to 0777. Which means all permissions to the owner, group, and others.
Why did we add a zero in 0777? It signals to Go that the number is not a decimal number but an octal number.
CSV means Comma-Separated Values. It is a file format designed to store data.
\nA CSV file is composed of several lines
Each new line is a data record
Each data record is composed of “fields”
A comma separates each field
The first line is often used as a header to describe what can be found in fields.
Here is an example CSV file :
\nage,genre,name\n23,M,Hendrick\n65,F,StephanyIn this section, we will see how to write data to a file. To immediately apply our knowledge, we will use a real use case: creating a CSV file from a slice.
\n// data-storage/file-save/writing/main.go \npackage main\n\nimport (\n "bytes"\n "fmt"\n "os"\n)\n\nfunc main() {\n s := [][]string{\n {"age", "genre", "name"},\n {"23", "M", "Hendrick"},\n {"65", "F", "Stephany"},\n }\n // Open the file or create it\n f, err := os.Create("myFile.csv")\n defer f.Close()\n if err != nil {\n fmt.Println(err)\n return\n }\n var buffer bytes.Buffer\n // iterate over the slice\n for _, data := range s {\n buffer.WriteString(fmt.Sprintf("%s,%s,%s\\n", data[0], data[1], data[2]))\n }\n n, err := f.Write(buffer.Bytes())\n fmt.Printf("%d bytes written\\n", n)\n if err != nil {\n fmt.Println(err)\n return\n }\n}We first create a slice of slices s. That’s the data we want to write into a CSV file.
We create a variable of type bytes.Buffer named buffer.
And then we iterate over s to add the data to the buffer. Each value is separated by a comma. We indicate that we want to add a new line by the characters \"\\n\".
When the buffer is ready, we write it to the file with the method Write. This method takes a slice of bytes as parameters and returns two values:
the number of bytes written to the file
an eventual error
We used in the previous program a data buffer. A buffer “is a region of physical memory storage used to store data temporarily while it is being moved from one place to another”2.
\nThe type bytes.Buffer can be very useful when you need to efficiently concatenate strings or manipulate slices of bytes.
We will use in the next section the following example use case :
\nA school has hired you.
The school needs a program to manage students and teachers :
\nget, add, update, delete a teacher
get, add, update, delete a student
In the following sections, we will see how to do that with different databases :
\n\nMySQL is an open-source relational database management system. In this section, we will see how to connect to a MySQL database and how to perform basic CRUD operations with Go.
\nGo is not shipped with a MySQL driver.
\nBut the standard library defines interfaces to manipulate a SQL database.
\nAt the time of writing, two drivers are proposed in the Go wiki.
\nI have chosen the most popular one on GitHub https://github.com/go-sql-driver/mysql, which is under the Mozilla Public License Version 2.0.
\n\n// data-storage/mysql/create/main.go \npackage main\n\nimport (\n "database/sql"\n "fmt"\n\n _ "github.com/go-sql-driver/mysql"\n)\n\nfunc main() {\n db, err := sql.Open("mysql", "root:root@tcp(localhost:8889)/school")\n if err != nil {\n fmt.Println(err)\n return\n }\n err = db.Ping()\n if err != nil {\n fmt.Println(err)\n return\n }\n}Here we are importing the standard package database/sql which defines all the functions needed to perform operations on a SQL database.
Then we use a blank import to register the driver github.com/go-sql-driver/mysql.
If you do not import a driver, you might encounter the following error :
\nsql: unknown driver "mysql" (forgotten import?)The function sql.Open accept two parameters :
The name of the driver
The Data Source Name (often called DSN)
This DSN parameter is a string where you define :
\nusername: root
password: root
the protocol to use to connect to the database: TCP
the host: localhost
the port: 8889
the database name: school
Following the documentation of the driver, the DSN as the following form :
\nusername:password@protocol(address)/dbname?param=valueThe function sql.Open returns a *sql.DB.
The method Ping will check if a connection is available; if not, it will create a new one.
The first thing you might want to do is to create a table into our database. Let’s write our SQL script :
\nCREATE TABLE `teacher` (\n `id` INT(11) NOT NULL AUTO_INCREMENT,\n `create_time` TIMESTAMP DEFAULT NULL,\n `update_time` TIMESTAMP DEFAULT NULL ON UPDATE CURRENT_TIMESTAMP,\n `firstname` VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,\n `lastname` VARCHAR(255) CHARACTER SET utf8 COLLATE utf8_general_ci DEFAULT NULL,\n PRIMARY KEY (`id`)\n) ENGINE=InnoDB;We create a table named teacher that will hold the data about each teacher of the school :
\nTheir first name
Their last name
Each teacher will have an id (our table’s primary key). We also add two technical columns that will save the date of creation (create_time) of the row and the last update date (update_time).
\nLet’s run this script.
\nFirst, we will save this SQL script into a file named “create_table.sql”. Our script will first load the script :
\nf, err := os.Open("create_table.sql")\nif err != nil {\n fmt.Println(err)\n return\n}\nb, err := ioutil.ReadAll(f)\nif err != nil {\n fmt.Println(err)\n return\n}In b, we have a slice of bytes that represent our query. Let’s execute it :
\nres, err := db.Exec(string(b))\nif err != nil {\n fmt.Println(err)\n return\n}We use string(b) to covert the slice of byte to a string because the Exec method takes a string as parameter.
Notice that two values are returned either a result (type sql.Result) or an error. sql.Result is a type interface with two methods :
LastInsertId() (int64, error)
RowsAffected() (int64, error)
We will just check that there are no errors; a create table does not affect existing rows nor insert data.
\n\nTo create a teacher, we will use an INSERT statement. We will use the Exec method :
// data-storage/mysql/insert/main.go \n\nfunc createTeacher(firstname string, lastname string, db *sql.DB) (int64, error) {\n res, err := db.Exec("INSERT INTO `teacher` (`create_time`, `firstname`, `lastname`) VALUES (NOW(), ?, ?)", firstname, lastname)\n if err != nil {\n return 0, err\n }\n id, err := res.LastInsertId()\n if err != nil {\n return 0, err\n }\n return id, nil\n}We have here a function createTeacher,that takes two strings (firstname, a lastname) and a pointer to a sql.DBas parameters.
We are passing to the Exec method three arguments.
db.Exec() is making a prepared statement.
Following the documentation of MySQL3, a prepared statement makes two distinct operations:
\nThe prepare stage: here, the client (our Go program) will send to the database server the template. The server will perform a syntax check of the request. It will also initialize resources for the next steps
The bind and execute stage: the client will then send to the server the values (in our case“John” and “Doe”). Then the server will build the query with those values and finally execute the query.
SQL Injection is a common vulnerability (SQL injections account for 20% of high severity vulnerabilities from 1988 to 2012
An SQL injection happens “when an attacker can insert a series of SQL statements into a ‘query’ by manipulating data input into an application.”
One solution (not the only one) is to use prepared statements in your code to prevent this kind of attack
To read data, we will use a SELECT statement.
To read one row into a database, you can use the method QueryRow. This method will return a pointer to an instance of sql.Row. The type sql.Row has a Scan method which we can use to inject the data retrieved from the DB into Go variables :
// data-storage/mysql/select/main.go \n\nfunc teacher(id int, db *sql.DB) (*Teacher, error) {\n teacher := Teacher{id: id}\n err := db.QueryRow("SELECT firstname, lastname FROM teacher WHERE id = ?", id).Scan(&teacher.firstname, &teacher.lastname)\n if err != nil {\n return &Teacher{}, err\n }\n return &teacher, nil\n}The function teacher will take an int and a pointer to a sql.DB as parameter.
The integer is the id of the teacher in the database.
This function will make a query to the database, with QueryRow.
The result is then returned via a *sql.Row.
We use then the Scan method.
This method will extract the data from the query result and copy it to the variables passed as parameters. Note that Go will handle for you the type conversion.
\nIn our example, we have created a Teacher type :
\ntype Teacher struct {\n id int\n firstname string\n lastname string\n}You cannot directly pass a Teacher variable to the Scan method, but you have to pass pointers :
Scan(&teacher.firstname, &teacher.lastname)Please pay attention to two common cases when you use QueryRow :
An error will be returned by the Scan method: the error sql.ErrNoRows :
\nvar ErrNoRows = errors.New("sql: no rows in result set")Scan will take the first row to be returned.
\n\nWhen you have a query that will return more than one row, you have to use the Query method and iterate over the rows.
\n// data-storage/mysql/selectMultiple/main.go \n\nrows, err := db.Query("SELECT id, firstname, lastname FROM teacher ")\nif err != nil {\n return nil, err\n}db.Query will return an element of type *sql.Rows. It represents a stream of results that is sent by the database server. When you are done reading the results of the query, call the method Close. We will use a defer statement to execute it at the end of our function.
defer rows.Close()\nteachers := make([]Teacher, 0)\nfor rows.Next() {\n // iterate over each row of the result set\n}You can see an idiomatic method to iterate over rows in the previous code listing.
\nWe use a for loop with rows.Next(). The idea is to create an empty slice of teachers (type Teacher) and then iterate over each row:
for rows.Next() {\n teacher := Teacher{}\n if err := rows.Scan(&teacher.id, &teacher.firstname, &teacher.lastname); err != nil {\n return nil, err\n }\n teachers = append(teachers, teacher)\n}When there are no more rows, the for loop will stop (because rows.Next() will return false). Scan is used to get the row data and set our struct fields.
The last instruction of the for loop will add the current teacher to theteachers slice.
At the end of this loop’s execution, we have a slice of teachers!
Here is the source code of the complete function we have to design :
\nfunc teachers(db *sql.DB) (*[]Teacher, error) {\n rows, err := db.Query("SELECT id, firstname, lastname FROM teacher ")\n if err != nil {\n return nil, err\n }\n defer rows.Close()\n teachers := make([]Teacher, 0)\n for rows.Next() {\n teacher := Teacher{}\n if err := rows.Scan(&teacher.id, &teacher.firstname, &teacher.lastname); err != nil {\n return nil, err\n }\n teachers = append(teachers, teacher)\n }\n return &teachers, nil\n}Without surprises, if you want to update a row (or several rows at once) you have to use the SQL UPDATE statement. The Exec method is optimal for this kind of operation.
It returns a sql.Result which is a type interface with two methods : LastInsertId() and RowsAffected().
The last method allows you to check how many rows in the table you have changed.
\nLet’s take an example you want to update the name of the teacher of id 1 :
\n// data-storage/mysql/update/main.go \n\nres, err := db.Exec("UPDATE teacher SET firstname = ? WHERE id = ?", "Daniel", 1)\nif err != nil {\n fmt.Println(err)\n // query was not a success; something went wrong\n}With our sql.Result value we can use the method RowsAffected(), to check that only one row was affected :
affected, err := res.RowsAffected()\nif err != nil {\n fmt.Println(err)\n return\n}\n\nif affected != 1 {\n fmt.Printf("Something went wrong %d rows were affected expected 1\\n", affected)\n} else {\n fmt.Println("Update is a success")\n}RowsAffected() return either an int64 or an error.
The type sql.DB represents a pool of one or more connections to the database. We can safely use it in concurrent programs.
Like MySQL, PostgreSQL is an open-source relational database management system. It was released in 1996.
\n\nTo connect to a database, we have to use a PostgreSQL driver.
\nThis example will use the driver https://github.com/lib/pq recommended by the Go repository wiki. It also seems to be a very popular repository that totalizes more than 4.000 stars and 70 contributors.
\nThe documentation of the driver can be found on godoc: https://godoc.org/github.com/lib/pq.
\n\nWe will use the function Open from the sql package :
// data-storage/postgresql/connection/main.go\npackage main\n\nimport (\n "database/sql"\n "fmt"\n\n _ "github.com/lib/pq"\n)\n\nfunc main() {\n db, err := sql.Open("postgres", "host=localhost port=5432 user=postgres dbname=school password=pg123 sslmode=disable")\n if err != nil {\n fmt.Println(err)\n return\n }\n err = db.Ping()\n if err != nil {\n fmt.Println(err)\n return\n }\n}You can notice that the first parameter of sql.Open is \"postgres\".
Internally Go will select and use the appropriate driver.
Then the second parameter is the DSN (Data Source Name).
This is a string that defines how to connect to the database. It is slightly different from the MySQL version, and I think it’s more expressive :
“host=localhost port=5432 user=postgres dbname=school password=pg123 sslmode=disable”
In a real application, you will activate sslmode (here, we have disabled it for test purposes), and in addition to that, it is not recommended to store those parameters in the source code. You should build the DSN strings from a config file’s values, for instance, or from environment variables.
\nOther parameters are available :
\nconnect_timeout - This is the timeout specified to wait for a connection to open. If you set it to zero or nothing, it will wait indefinitely.
sslcert, sslkey, sslrootcert: are used for SSL support4
PostgreSQL uses the SQL language but compared to MySQL; the CREATE TABLE syntax is not the same. Postgres types are not the same as MySQL ones.
\nHere is a script to create a similar table :
\nCREATE TABLE public.teacher\n(\n id serial,\n create_time timestamp without time zone,\n update_time timestamp without time zone,\n firstname character varying(255),\n lastname character varying(255),\n PRIMARY KEY (id)\n);Some specificities appear:
\nThe type serial is used for the primary key
You can add timezones to the timestamps (in MySQL, you can specify the timezone per connection, by default, it will be the server’s timezone).
The type named character varying(255) can be used to store strings.
To create the table, we will execute the script with the Exec method :
// load SQL script\n// use os.Open("create_table.sql") to open the script\n// then ioutil.ReadAll to load it's content\n// execute the script\nres, err := db.Exec(string(b))\nif err != nil {\n fmt.Println(err)\n return\n}\n// success, table created !The syntax is similar to the one used with a MySQL database.
\n\nFor insert (or update), the appropriate method to use is Exec for MySQL databases. We can use this method for PostgreSQL databases, but it is not optimal.Exec will return a Row, which defines the method LastInsertId. This method will make the request
SELECT LAST_INSERT_ID();With PostgreSQL, this method will return an error (this function does not exists). Instead, we can add to the SQL INSERT query RETURNING id;.
The complete SQL request is :
\nINSERT INTO public.teacher (create_time, firstname, lastname)\nVALUES (NOW(),$1, $2)\nRETURNING id;PostgreSQL do not support the \"?\" character, instead you need to use placeholders ($1,$2)
We will have to use the method QueryRow followed by a call to the Scan method to get the id inserted :
// data-storage/postgresql/insert/main.go \n\nfunc createTeacher(firstname string, lastname string, db *sql.DB) (int, error) {\n insertedId := 0\n err := db.QueryRow("INSERT INTO public.teacher (create_time, firstname, lastname) VALUES (NOW(),$1, $2) RETURNING id;", firstname, lastname).Scan(&insertedId)\n if err != nil {\n return 0, err\n }\n if insertedId == 0 {\n return 0, errors.New("something went wrong id inserted is equal to zero")\n }\n return insertedId, nil\n}The function teacher takes three arguments (the firstname, the lastname and a pointer to a sql.DB).
A pointer to the variable insertedId is passed as argument to the Scan method.
Go will update the value of insertedId with the result of the SQL query (which is the id inserted).
The call to db.QueryRow(..).Scan(...) returns an error if something went wrong.
We have created a type struct Teacher that will hold the data related to one teacher in our database.
\ntype Teacher struct {\n id int\n firstname string\n lastname string\n}Here is an example function to retrieve a teacher in the database
\n// data-storage/postgresql/select/main.go \n\nfunc teacher(id int, db *sql.DB) (*Teacher, error) {\n teacher := Teacher{}\n err := db.QueryRow("SELECT id, firstname, lastname FROM teacher WHERE id > $1 ", id).Scan(&teacher.id, &teacher.firstname, &teacher.lastname)\n if err != nil {\n return &teacher, err\n }\n return &teacher, nil\n}QueryRow will return a single row from the database.
If your query returns several rows only the first row will be treated.
The Scan function will extract the values from the returned row.
To read multiple rows, you should use the Query method :
// data-storage/postgresql/select-multiple/main.go \n\nrows, err := db.Query("SELECT id, firstname, lastname FROM teacher")Query returns either a pointer to a sql.Rows or an error.
You have to call rows.Close() to release the connexion :
// close the connexion at the end of the function\ndefer rows.Close()Then, we iterate over the rows with rows.Next() :
\nfor rows.Next() {\n // treat one row\n}Inside the for loop, we will extract the data from the current row with the method Scan :
err := rows.Scan(&id, &firstname, &lastname)The function convertAssign (in the package sql) will be called :
func convertAssign(dest, src interface{}) error {\n //...\n}dest is a pointer to our destination variable.src is the source, what the database driver has returned.
destand src.The function totalize 200+ lines of code. We will not reproduce its whole content here, but just the first lines// extract of function convertAssign (line 209, src/database/sql.go)\nfunc convertAssign(dest, src interface{}) error {\n // Common cases, without reflect.\n switch s := src.(type) {\n case string:\n switch d := dest.(type) {\n case *string:\n if d == nil {\n return errNilPtr\n }\n *d = s\n return nil\n case *[]byte:\n if d == nil {\n return errNilPtr\n }\n *d = []byte(s)\n return nil\n case *RawBytes:\n if d == nil {\n return errNilPtr\n }\n *d = append((*d)[:0], s...)\n return nil\n }\n //...\n}The function is composed of a big switch statement. We switch over the type of src. The type is retrieved with the statement :
s := src.(type)If the src variable is a string we will then extract the type of the dest argument :
d := dest.(type)Another switch case statement is launched.
\nIf src is a string, the dest is a *[]byte. The following code snippet is executed :
if d == nil {\n return errNilPtr\n}\n*d = []byte(s)\nreturn nilThe function will test that the destination pointer is not nil. If not, the value of dest will be changed to []byte(s) :
*d = []byte(s)Like for MySQL, the appropriate method is Exec. This method will return a type that implements sql.Result. res.RowsAffected() will return the number of rows in the table that was affected by the UPDATE statement. You can control the impact of your request on the table by controlling this value :
// data-storage/postgresql/update/main.go \n\nres, err := db.Exec("UPDATE teacher SET firstname = $1 WHERE id = $2", "Daniel", 1)\n// check that there is no error\n// get the number of affected rows\naffected, err := res.RowsAffected()\n//...For a full listing example see [subsec:Update-row(s)]. The only difference with MySQL is the format of placeholders. Instead of \"?\" you should use \"$n\" (where n is a
MongoDB was created in 2007 by the 10gen company. A proprietary system that was part of a Platform As A Service application, MongoDB has been released by its creators as an open-source database.
\nMongoDB is part of the NoSQL trend.
\nThe term NoSQL was created in 2009 during a meetup about databases by Johan Oskarsson5.
\nThere are no rigorous definitions of NoSQL databases, but they share two common characteristics:
\nThey are document-oriented (they store documents instead of rows in tables)
They are designed to be highly scalable
This section will focus on basic MongoDB operations (CRUD). But first, let’s study some important vocabulary :
\n\nThis is a JSON string that we will store in MongoDB with a specific format the BSON
\nThe binary representation of JSON used by MongoDB to store documents
\nA set of documents (there is no obligation to store document of the same type in collections)
\nA set of collections
\nTo connect and make operations on our database, we will use MongoDB’s official driver: github.com/mongodb/mongo-go-driver.
\n\n// data-storage/mongodb/official-driver/connection/main.go\npackage main\n\nimport (\n "context"\n "fmt"\n "time"\n\n "github.com/mongodb/mongo-go-driver/mongo"\n)\n\nfunc main() {\n client, err := mongo.NewClient(`mongodb://username:password@localhost:27094/db`)\n if err != nil {\n panic(err)\n }\n //...\n\n}We create a mongo client with the function NewClient that takes the database connection string. This string has the following format :
mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]You can specify several hosts to create the client. Note that I have specified a database at the end of the connection string in my example. This is not mandatory. For some cloud database providers, it’s mandatory to specify the database in the connection string for authentification purposes. Those cloud providers share MongoDB servers between their clients, and authentification is handled “by database”.
\nAfter creating the client, we will connect to the database :
\nctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)\ndefer cancel()\nerr = client.Connect(ctx)\nif err != nil {\n panic(err)\n}The first thing we are doing in the previous code snippet is to create a context for the connection. Context is a standard Go package (I have dedicated a chapter to this package) that will define a specific timeout for the connection (here 3 seconds). If the program does not succeed in making the connection, it will panic with the following message :
\npanic: context deadline exceededAfter connecting to the database, you can ping the server with the following code snippet :
\nerr = client.Ping(ctx, nil)\nif err != nil {\n panic(err)\n}Data is stored in documents. A document is a JSON string. Documents are stored in collections. Hence, the first step is to retrieve the collection in the database :
\ncollection := client.Database("db").Collection("school")Note that if the collection does not exist, it will be created automatically.
\n// data-storage/mongodb/official-driver/create/main.go \n\nres, err := collection.InsertOne(context.Background(), bson.M{"hello": "world"})\nif err != nil {\n panic(err)\n}\nid := res.InsertedID\nfmt.Println(id)Here we are using the method InsertOne. This method takes a context as its first argument (here again, you can define a timeout) and as second argument a document. The package does not force you to use a specific type for the second argument :
func (coll *Collection) InsertOne(ctx context.Context, document interface{}, opts ...*options.InsertOneOptions) (*InsertOneResult, error)(the second argument type is the empty interface).
\nIn the example, we are passing to the function :
\nbson.M{"firstname":"John", "lastname":"Doe", "create_time":time.Now()}bson is a subpackage from the driver. The M type is a type alias for a map[string]interface{}.Here is the document inserted in the database :
{\n "_id": {\n "$oid": "5c0919b285d8ae1a8afe6c80"\n },\n "lastname": "Doe",\n "create_time": {\n "$date": "2018-12-06T12:44:34.168Z"\n },\n "firstname": "John"\n}MongoDB generates the id. Then you can see that the date property is handled specifically (to use later in filters).
\n\nTo retrieve a document by one of its property, we will create a filter variable :
\nfilter := bson.D{{"firstname", "Jeanne"}}bson is a sub-package of the Go MongoDB Driver. It defines utils to parse and generate BSON. The type bson.D represents a BSON document. (D stands for document). Internally bson.D is a slice of primitive.E which represents an element.primitive.E is a type struct with two exported fields Key (type string) and Value (type empty interface). Here we filter documents that have the property “firstname” set to “Jeanne”.
Then we will prepare the variable that will hold our result :
\ntype Teacher struct {\n Firstname string\n Lastname string\n}Then we create of type Teacher to hold the query result :
result := Teacher{}To query the database, we will use an instance of Collection :
\n// data-storage/mongodb/official-driver/read/main.go \n\ncollection := client.Database(databaseName).Collection(collectionName)\nerr = collection.FindOne(context.Background(), filter).Decode(&result)\nif err != nil {\n panic(err)\n}\n// One document was found\nfmt.Printf("%+v", result)First, we retrieve the collection (we select the database and then the collection).
The database name and the collection name are stored inside variables.
With the method FindOne the server will return only one result that matches the provided filter. The method takes two parameters :
A context
The filter.
The method FindOne returns a pointer to an element of type SingleResult.
Then we make a call directly to the method SingleResult. (we could have done that in two different steps, but the code is simpler that way).
What if there is no result? In this case, Decode will return an error. Our program will panic :
\npanic: mongo: no documents in resultEach document in MongoDB has a property _id. We can compare it to a primary key in relational database engines.
\nThis field has a special type called ObjectId which has the following properties :
It’s composed of 12 bytes.
It’s composed of 4 parts (the number of seconds since UNIX epoch, an identifier of the machine, the id of the process, and a counter)
The client generates it.
Before building the query, we have to create a variable of type ObjectId.
We are looking for the document with _id equal to \"5c091d3b734016209db89f76\"
// data-storage/mongodb/official-driver/readById/main.go\n\nimport "github.com/mongodb/mongo-go-driver/bson/objectid"\n\n//...\noid , err := objectid.FromHex("5c091d3b734016209db89f76")\nif err != nil {\n panic(err)\n}The variable oid is ready to be used in a query. Let’s create the filter variable to retrieve our document :
filter := bson.M{"_id": oid}Then we just have to call the FindOne method :
\nresult := Teacher{}\nerr = collection.FindOne(context.Background(), filter).Decode(&result)\nif err != nil {\n panic(err)\n}\nfmt.Printf("%+v", result)\n// {Id:ObjectID("5c091d3b734016209db89f76") Firstname:Jeannoti Lastname:Doe CreateTime:2018-12-06 13:59:39.338 +0100 CET}We create a brand new variable of type Teacher to hold our query result.
The result properties are hydrated by the Decode method (we pass to Decode a pointer to the variable result).
A MongoDB query consists of a JSON-like object. To build a query, you have to use “query selectors”. Those selectors start with the dollar symbol, for instance $eq means equal, $neq not equal. This section will review some of them; then, we will build a query.
\n| Selector | \nDef. | \nQuery Example | \nExplanation. Match all documents... | \n
|---|---|---|---|
| $eq | \nequal | \n{ firstname: { $eq: John } } | \nthat have a property firstname equal to John | \n
| $ne | \nnot equal | \n{ lastname: { $ne: Doe } } | \nthat have a property lastname not equal to Doe | \n
| $gt | \ngreater than | \n{ age: { $gt: 12 } } | \nthat have a property age strictly greater than 12 | \n
| $gte | \ngreater than or equal | \n{ age: { $gte: 42 } } | \nthat have a property age greater or equal than 12 | \n
| $lt | \nless than | \n{ monthlyIncome: { $lt: 1000 } } | \nthat have a property monthlyIncome less than 1000 | \n
| $lte | \nless than or equal | \n{ monthlyIncome: { $lte: 1000 } } | \nthat have a property monthlyIncome less equal to 1000 | \n
| $in | \nin array | \n{ age: { $in: [20,42] } } | \nthat have a property age equal to 20 or 42 | \n
| $nin | \nnot in array | \n{ age: { $nin: [20,30] } } | \nthat have a property age not equal to 20 or 30 | \n
| Selector | \nDefi. | \nQuery Example | \nExplanation. Match all documents... | \n
|---|---|---|---|
| $and | \nAND | \n{ $and: [{ age: { $gt: 60 } },{ firstname: { $eq: John} } ] } | \nThat have property age greater than 60 and firstname equal to John | \n
| $or | \nOR | \n\n | That have the age property greater than 12 or less than 3 | \n
| $nor | \nNOR | \n{ $nor: [{ age: { $gt: 12 } },{ age: { $lt: 3 } } ] } | \nthat don’t have age greater than two and age less than 3 | \n
| $not | \nNOT | \n{ $not: [{ age: { $gt: 12 } }] } | \nthat don’t have the property age greater than 12 | \n
Let’s create a query for our school collection. We want to retrieve all teachers that have John as firstname and Doe as lastname :
\n{\n "$and": [{\n "firstname": {\n "$eq": "John"\n }\n },\n {\n "lastname": {\n "$eq": "Doe"\n }\n }\n ]\n}Here we use the $and logical selector. The value of $and is an array of query clauses.
Here the array has two elements: 2 queries that will be joined by a logical AND. We are looking for teachers that have a firstname equal to John and a last name equal to Doe.
\nLet’s build the two query clauses :
\n// data-storage/mongodb/official-driver/query-advanced/main.go\n\n//.. \n\nq1 := bson.M{"firstname": bson.M{"$eq": "John"}}\nq2 := bson.M{"lastname": bson.M{"$eq": "Doe"}}q1 and q2 are of type bson.M which is internally a map with a string key and a value of type empty interface. Let’s analyze the q11 variable : the key is \"firstname\" and the value is another variable of type bson.M with a key equal to $eq and the value is \"John\". To join our two queries, we have to create a BSON array. The appropriate type to use is bson.A (A stands for array) :
clauses := bson.A{q1, q2}Then we define the main query with the $and logical selector with an element of type bson.M:
filter := bson.M{"$and":clauses }The query is ready to be sent. We will use the method Find of the Collection type :
cur, err := collection.Find(context.Background(), filter)\nif err != nil {\n panic(err)\n}Classically Find returns either a Cursor or an error. A Cursor is a pointer to the results of a query. We can iterate over a cursor by calling the Next method. This method will return true if there is no error and if a next result exists.
Each result represents a document. The document data is obtained by calling the method DecodeBytes :
defer cur.Close(context.Background())\nfor cur.Next(context.Background()) {\n raw, err := cur.DecodeBytes()\n if err != nil {\n panic(err)\n }\n fmt.Printf("%s\\n", raw)\n}\nif err := cur.Err(); err != nil {\n panic(err)\n}The first thing to do is to call defer cur.Close(context.Background()) to close the cursors when the surrounding function returns. Then we iterate over the cursor with a for loop. Inside the for loop, we are retrieving the raw document value, and we store it inside the raw variable.
In the case of an error, we will stop the for loop, and the next statement will be executed. We will retrieve the error by calling the Err method. The program will panic in the case of an error.
Cursors can timeout. It means that if you take too much time to iterate over the results, the cursor might be closed by the server after a certain period of inactivity. You can specify the type of cursor you want to use in your query. Cursors have two main types :
\nTailable : the cursor will remain open even if the last data has been retrieved, we can reuse the cursor.
Non Tailable : the cursor will close automatically after retrieving the last data.
By default, cursors are non-tailable.
\n\nTo update a document in MongoDB, you have to provide two BSON objects :
\nA query that will find documents to update. It’s the filter.
The modifications to apply to the found document.
Let’s take an example. We want to update the first document that has the property firstname set to John and replace the property lastname by CoffeeBean.
\nThe filter is :
\n{\n "firstname": {\n "$eq": "John"\n}We can simplify this query:
\n{"firstname":"John"}The modification to apply is :
\n{\n "$set": {\n "lastname": "CoffeeBean"\n}Let’s create the two BSON objects!
\n// data-storage/mongodb/official-driver/update/main.go \n//...\nfilter := bson.M{"firstname": "John"}\nupdate := bson.M{"$set": bson.M{"lastname": "CoffeeBean"}}Then we will use the method FindOneAndUpdate that takes as argument a context, the filter bson, and the update bson :
res := collection.FindOneAndUpdate(context.Background(), filter, update)This method returns a pointer to a DocumentResult. We will use the method Decode to parse the returned document. If our filter has returned 0 document, then no updates will be performed, and no document returned. In this case, an error is returned.
resDecoded := Teacher{}\nerr = res.Decode(&resDecoded)\nif err != nil {\n panic(err)\n}\nfmt.Printf("%+v", resDecoded)Will output:
\n{Id:ObjectID("5c091d3b734016209db89f76") Firstname:John Lastname:CoffeeBean CreateTime:2018-12-06 23:59:39.338 +0100 CET}ElasticSearch is an open-source search engine software. Shay Banon originally developed it, and the first release took place in 2010. In 2018 Elasticsearch is in the top 10 of the most popular databases in the Stackoverflow developer survey
ElasticSearch is mostly used to develop search engines. It is developed in Java, and it is based on Apache Lucene. Apache Lucene is developed by the Apache Software Foundation and is a“high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform”
The search engine’s basic unit is not a row but a document. An elastic search document is a JSON object. The structure of each document is not fixed. It’s a schemaless database.
\nIn this section, we will use the client library https://github.com/olivere/elastic under MIT license.
\n\nLet’s define some important terms :
\nThe action of storing data into ElasticSearch. Data are organized in documents.
\nA JSON object that represents data. A document has a type that belongs to an index.
\nAn ElasticSearch cluster can contain multiple indices (plural of index). From a relational perspective, an index can be seen as a table.
\nEach document stored in ElasticSearch has a specific type. If you want to store the firstname, the lastname, the date of arrival of a teacher, you will create a document of type teacher.
\nA type is composed of fields. For instance, the type teacher will have the fields firstname and lastname
\nNote that types are in the process of being deprecated. The notion of type will disappear from the ElasticSearch API in the future. 6.
\nWe will follow the recommended approach. Which is to create a type _doc that has a field type
\n\nElasticSearch is designed to be run on a cluster of machines. A machine is called a node. A cluster has a master node. The master node is not fixed; the master capacity can be attributed to another node. The Go client has been designed to automatically detect new nodes and remove nodes no longer used in a cluster.
\nTo create the client, you will have to pass the address (host + port) of nodes of your cluster :
\n// data-storage/elasticsearch/create-index/client/main.go \n//...\nclient, err := elastic.NewClient(elastic.SetURL("http://127.0.0.1:9200"))If you have other nodes in your cluster, just add the other address to the functionSetURL :
\nclient, err := elastic.NewClient(elastic.SetURL("http://127.0.0.1:9200", "http://127.0.0.1:9300"))This client is designed to be used in your whole application. With this client, you can perform CRUD operations. Note that you can also do those operations with a simple HTTP request.
\n\nBefore storing any document, we have to create an index. Documents in ElasticSearch are stored on indexes. An index has a mapping that defines a type with fields. We will name our index “school”.
\nYou need first to define your mapping :
\nconst Mapping = `{\n "mappings": {\n "_doc": {\n "properties": {\n "type": { "type": "keyword" },\n "firstname": { "type": "text" },\n "lastname": { "type": "keyword" },\n "create_time": { "type": "date" },\n "update_time": { "type": "date" }\n }\n }\n }\n}`To create the index, you have to make an http request :
\nPUT http://127.0.0.1:9200/{indexName}\n{\n"mappings" : {\n //...\n }\n}To make a request with Go, you can use the following code snippet :
\n// data-storage/elasticsearch/create-index/rest/main.go \n//...\nclient := &http.Client{}\n\n// prepare the request\nrequest, err := http.NewRequest("PUT", "http://127.0.0.1:9200/school",strings.NewReader(Mapping))\nrequest.Header.Add("Content-Type", "application/json")\n\n// execute the request\nresponse, err := client.Do(request)\n if err != nil {\n log.Fatal(err)\n } else {\n defer response.Body.Close()\n contents, err := ioutil.ReadAll(response.Body)\n if err != nil {\n log.Fatal(err)\n }\n fmt.Printf("%s", contents)\n }\n}It will output the following :
\n{"acknowledged":true,"shards_acknowledged":true,"index":"school"}Meaning that your index has been created.
\n\n// data-storage/elasticsearch/create-index/client/main.go \n//...\n\ncreateIndex, err := client.CreateIndex("school").BodyString(Mapping).Do(ctx)\nif err != nil {\n // Handle error\n fmt.Println(err)\n return\n}\nif !createIndex.Acknowledged {\n // Not acknowledged\n fmt.Println("not acknoledge !")\n return\n}\nfmt.Println("OK")With ElasticSearch, the endpoint to create the document is also the endpoint to update the document.
\n\nTo create a document you just have to make a PUT request at the url \"http://127.0.0.1:9200/{indexName}/{typeName}/{id}\", where {id} it the id of the document you want to add in the index {indexName}.
In our case :
\n_doc
\n1
\nschool
\nPUT http://127.0.0.1:9200/school/_doc/42\n{\n "type":"teacher",\n "firstname":"John",\n "lastname":"Doe",\n "create_time":"2018-09-10 00:00:00"\n}Here the Go code snippet to perform our insert :
\n// data-storage/elasticsearch/insert/rest/main.go \n\n//...\n\nconst NewTeacher = `\n{\n "type":"teacher",\n "firstname":"John",\n "lastname":"Doe",\n "create_time":"2018-09-10T00:00:00"\n}`\n\nconst ClusterUrl = "http://127.0.0.1:9200"\nconst IndexName = "school"\nconst Username = "myUsername"\nconst Password = "myPassword"\nconst TypeName = "_doc"\nconst ObjectId = 42\n\nfunc main() {\n\n url := fmt.Sprintf("%s/%s/%s/%d", ClusterUrl, IndexName, TypeName, ObjectId)\n client := &http.Client{}\n request, err := http.NewRequest("PUT", url, strings.NewReader(NewTeacher))\n request.Header.Add("Content-Type", "application/json")\n // if your cluster requires authentification\n token := base64.StdEncoding.EncodeToString([]byte(fmt.Sprintf("%s:%s", Username, Password)))\n request.Header.Add("Authorization", fmt.Sprintf("Basic %s", token))\n\n response, err := client.Do(request)\n if err != nil {\n log.Fatal(err)\n } else {\n defer response.Body.Close()\n contents, err := ioutil.ReadAll(response.Body)\n if err != nil {\n log.Fatal(err)\n }\n fmt.Printf("%s", contents)\n }\n}The previous code snippet will output (if no errors) a JSON object :
\n{\n "_index": "school",\n "_type": "_doc",\n "_id": "42",\n "_version": 1,\n "result": "created",\n "_shards": {\n "total": 2,\n "successful": 2,\n "failed": 0\n },\n "_seq_no": 3,\n "_primary_term": 1\n}You can see that our document has been successfully created on two shards. The document has version number 1. You will see that this property will increment each time you update your document.
\n\nThe first step is to create a type struct Teacher :
\n// data-storage/elasticsearch/insert/client/main.go \n\ntype Teacher struct {\n Firstname string `json:"Firstname"`\n Lastname string `json:"Lastname"`\n CreateTime time.Time `json:"create_time"`\n UpdateTime time.Time `json:"update_time"`\n}Then you can create your new element to insert into ElasticSearch :
\nteacher := Teacher{firstname: "John", lastname: "Doe", createTime: time.Now()}Then the client API is pretty straightforward :
\nput, err := client.Index().\n Index("school").\n Type("_doc").\n Id("42").\n BodyJson(teacher).\n Do(ctx)\nif err != nil {\n // Handle error\n panic(err)\n}You have to precise the Index name, the type, the Id and finally the JSON document with the method BodyJson.
The put variable is of type *elastic.IndexResponse. We can check afterward if the document has been created or updated :
if put.Result == "updated" {\n fmt.Println("document has been updated")\n fmt.Printf("version N. : %d\\n", put.Version)\n}\nif put.Result == "created" {\n fmt.Println("document has been created")\n}To get a document with its id, you must perform an HTTP GET request. For instance, if you want to retrieve the document of type _doc in the index school with the id 42, you have to launch the following request :
\nGET http://127.0.0.1:9200/school/_doc/42The first thing to do is to build the URL :
\nurl := fmt.Sprintf("%s/%s/%s/%d?pretty=true", ClusterUrl, IndexName, TypeName, ObjectId)Here we are using Sprintf to generate the request URL.ClusterUrl, IndexName, TypeName, ObjectId are constants that are defined at the beginning of the file. Note that we added the parameter ?pretty=true to our request to inform the server that we want the JSON result to be returned with indentation.
Then we have to create a standard HTTP client and build our request :
\n// data-storage/elasticsearch/read/by-id/rest/main.go \n//...\n\nclient := &http.Client{}\nrequest, err := http.NewRequest("GET", url, nil)\n// add the authorization header\nrequest.Header.Add("Authorization", fmt.Sprintf("Basic %s", token))The next step is to send the request and get the result :
\nresponse, err := client.Do(request)\nif err != nil {\n log.Fatal(err)\n} else {\n defer response.Body.Close()\n contents, err := ioutil.ReadAll(response.Body)\n if err != nil {\n log.Fatal(err)\n }\n fmt.Printf("%s", contents)\n}The previous code snippet will output :
\n{\n "_index" : "school",\n "_type" : "_doc",\n "_id" : "42",\n "_version" : 3,\n "found" : true,\n "_source" : {\n "type" : "teacher",\n "firstname" : "John",\n "lastname" : "Doe",\n "create_time" : "2018-09-10T00:00:00"\n }\n}You can then parse this JSON string (with json.Unmarshall) and extract the value of the boolean property “found” and test it to inform the user of your application that a result has been retrieved.
You can use the Get method :
// data-storage/elasticsearch/read/by-id/client/main.go \n//...\nget, err := client.Get().\n Index(IndexName).\n Type(TypeName).\n Id(ObjectId).\n Do(ctx)\nif err != nil {\n // error\n // index not found\n // document with this id not found\n // ...\n panic(err)\n}\nif get.Found {\n // document was found\n}Here an element of type *elastic.GetResult will be returned. The property Found allow you to check if a document was retrieved. You can also get the Version, the Id...etc.
ElasticSearch is a search engine and a good one. We can perform the search operations on indices by using the rest API. Our goal is not to cover all the engine search features but to show you how to make it work with Go. Let’s get to it!
\n\nElasticSearch exposes a REST GET endpoint designed for search. For instance, if you want to find all the teachers that have a firstname equals to “John” we can use the following request:
\nGET school/_search\n{\n "query": {\n "bool": {\n "must": {\n "match": {\n "firstname": "Danny"\n }\n },\n "filter": {\n "match": {\n "type": "teacher"\n }\n }\n }\n }\n}We have created the constant SearchJSON that will hold the JSON query addressed to the server. You might want to build it from the user input in a real case. Be careful with potential query injections!
// data-storage/elasticsearch/read/search/rest/main.go \n// ...\n\nurl := fmt.Sprintf("%s/_search?pretty=true", ClusterUrl)\nclient := &http.Client{}\nrequest, err := http.NewRequest("GET", url, strings.NewReader(SearchJSON))\nrequest.Header.Add("Content-Type", "application/json")\nrequest.Header.Add("Authorization", fmt.Sprintf("Basic %s", token))Then we launch the request, exactly like in the previous section...
\nIf there are no results, the result JSON returned by the ElasticSearch is the following one :
\n{\n "took" : 1,\n "timed_out" : false,\n "_shards" : {\n "total" : 4,\n "successful" : 4,\n "skipped" : 0,\n\n "failed" : 0\n },\n "hits" : {\n "total" : 0,\n "max_score" : null,\n "hits" : [ ]\n }\n}If you have a result (or several), you will have to handle the following JSON :
\n{\n "took" : 2,\n "timed_out" : false,\n "_shards" : {\n "total" : 4,\n "successful" : 4,\n "skipped" : 0,\n "failed" : 0\n },\n "hits" : {\n "total" : 1,\n\n "max_score" : 0.9808292,\n "hits" : [\n {\n "_index" : "school",\n "_type" : "_doc",\n "_id" : "72",\n "_score" : 0.9808292,\n "_source" : {\n "type" : "teacher",\n "firstname" : "Danny",\n "lastname" : "Doe",\n "create_time" : "2018-09-10T00:00:00"\n }\n }\n ]\n }\n}The Go client has a lot of methods to handle queries to ElasticSearch, but also aggregation. Its API is very rich. Hence we will not go deep into the details for this section.
\nHere is an example of how to build a query that will return all the documents that have a property firstname equal to Danny :
\n// data-storage/elasticsearch/read/search/client/main.go \n// ...\n\nq := elastic.NewBoolQuery()\nq.Must(elastic.NewMatchQuery("firstname", "Danny"))This query can then be passed to the client :
\nres, err := client.Search().\n Index(IndexName). // search in index "twitter"\n Query(q).\n Pretty(true). // pretty print request and response JSON\n Do(ctx) // execute\nif err != nil {\n // do something when an error is raised\n}\n// Success! the query returned 0 or more resultsTe variable res is of type *elastic.SearchResult. This type has interesting properties :
the duration of the request (type int64)
\nthe results (type *elastic.SearchHits)
\nA boolean that will be true if your query has timed out
\nLet’s see how to use res.Hits :
\nif res.Hits.TotalHits > 0 {\n fmt.Printf("Found %d hits\\n", res.Hits.TotalHits)\n // Iterate through results\n for _, hit := range res.Hits.Hits {\n var t Teacher\n err := json.Unmarshal(*hit.Source, &t)\n if err != nil {\n panic("impossible to deserialize")\n }\n // enjoy your fresh instance of Teacher!\n fmt.Printf("Teacher found firsname %s - lastname : %s\\n", t.firstname, t.lastname)\n }\n} else {\n // No results\n}You first begin by checking that the TotalHits property is greater than zero. If true, it’s worth examining the results...
\nThen with a for loop, we iterate over the Hits, which are a slice of *SearchHit elements. With *hit.Sourcewe access to the _source json property that is returned by the server :
//...\n"_source" : {\n "type" : "teacher",\n "firstname" : "John",\n "lastname" : "Doe",\n "create_time" : "2018-09-10T00:00:00"\n}\n//...We are converting the JSON String into a variable of type Teacher with json.Unmarshal. The variable t is ready to be used afterward by your program.
What is a file mode?
How to create a connection to a SQL database?
Which method can you use to execute INSERT and UPDATE queries on a SQL database?
Which method can you use to parse the results of an SQL request?
What is a file mode?
\nHow to create a connection to a SQL database?
\nInstall the appropriate driver
Call sql.Open
Which method can you use to execute INSERT and UPDATE queries on a SQL database?
\nWhich method can you use to parse the results of an SQL request?
\nTo create a file use os.Create
To create a file and directly write to it, you can use ioutil.WriteFile
On UNIX systems, each file has a set of permissions
You can change the permissions of a file with the function os.Chmod
After installing a driver, you can use SQL databases in your program
Do not forget to import your driver with a blank import
\nsql.Open can be used to open a new connection to a database
It will return a pointer to a sql.DB that has the following methods
Exec : to execute raw queries (inserts, update,...)
QueryRow : to select a single row
Query : to select multiple rows
You can convert query results to variables with the method Scan
You can use MongoDB and Elasticsearch databases thanks to extensive open-source clients.
Octal is a numeral system (like decimal hexadecimal or binary) is has a base 8. Each digit in an octal number are equal to 0,1,2,3,4,5,6 or 7↩︎
Source : https://en.wikipedia.org/wiki/Data_buffer↩︎
https://dev.mysql.com/doc/apis-php/en/apis-php-mysqli.quickstart.prepared-statements.html↩︎
See https://www.postgresql.org/docs/9.0/libpq-ssl.html↩︎
NoSQL means Not Only SQL↩︎
See https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html↩︎
Previous
\n\t\t\t\t\t\t\t\t\tDates and time
\n\t\t\t\t\t\t\t\tNext
\n\t\t\t\t\t\t\t\t\tConcurrency
\n\t\t\t\t\t\t\t\t