What is a log?
How to use the standard logging package?
How to set up and use logrus?
What are log levels?
Log
Log level
Severity
Syslog
Deamon
A log is a structured and parsable message produced by an application installed on a system. It’s the developer’s task to add logs to its application.
\nAn application without bugs does not exist; even if you apply the best practices, you will have to face at a certain point of time a failure. A failure can be recoverable, meaning that your system can handle it and continue to work. It can also be unrecoverable. In the last case, your application will stop working.
\n\nImagine the following situation: your phone starts to ring in the middle of the night. The CEO is at the other end of the line. He informs you that the site you have developed is down. Customers cannot place orders. You have to make it work now. You connect to the machine, restart the program, but it won’t restart. With stress reaching high levels, you will try to figure out why.
\nWith logs, you can investigate and discover the origin of an error. Without logs, it can be far more difficult. But even with logs, your task will not be easy.
\nThe difficulty will decrease if you know :
\nWhen your system has failed
Where the error has been raised
Which user/operation was involved when the system failed.
The worst thing that can happen to an application is to produce errors, but those errors stay unnoticed, and bugs are still there. A solid logging habit allows you to monitor your application and maintain it in good health.
\n\nHere is an example of logs :
\n2018/10/14 20:08:08 System is starting\n2018/10/14 20:08:08 Retrieving user 12\n2018/10/14 20:08:08 Compute the total of invoice 1262663663As you can see, we have a timestamp followed by a string, which is the message.
\n\nIn the standard library the log package implements a very simple yet powerful logger.
The first thing to do is to define the Output of your logger. If you omit to define it, it will be the standard error. In the example, we are going to log into a file :
\n// logging/main.go \n\nf, err := os.OpenFile("app.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)\nif err != nil {\n log.Fatalf("error opening file: %v", err)\n}\ndefer f.Close()\nlog.SetOutput(f)We use the package os to open our file (os.OpenFile) the call to this function takes the following parameters :
The name of the file : \"app.log\"
The flags describe how the file has to be opened by the system. In our case, we want to append data to it (O_APPEND), create it if it does not exist (O_CREATE) and open it in write-only mode (O_WRONLY)
The set of permissions.0644 means that the user can read and write, the group and others can write. If the file does not exist, the set of permissions will be applied.
We are calling defer.Close(), to close the file when the main function has returned.
Then we set the output of the standard logger to this opened file (log.SetOutput takes an io.Writer)
We are ready to log :
\n// logging/main.go \nlog.Println("System is starting")\nlog.Println("Retrieving user 12")\nlog.Println("Compute the total of invoice 1262663663")To log, you can use the functions Print, Printf, Println, Fatal, Fatalf...
log.Print(\"System failure caused by lost DB connexion\") will output to the standard logger :
2018/10/14 20:08:08 System failure caused by lost DB connexionIf you call log.Println(\"xyz\") the execution result of fmt.Sprintln(\"xyz\") will be printed to the standard logger :
log.Println("System failure caused by lost DB connexion")log.Printf is interesting. It allows you to add to your message formatted variables. You can use format specifiers to tell Go how it should output the variable.
dbIp:="192.168.210.12"\ndbPort := 3306\nlog.Printf("System failure caused by lost DB connexion to host %s at port %d", dbIp,dbPort)Will output the following :
\n2018/10/16 13:29:07 System failure caused by lost DB connexion to host 192.168.210.12 at port 3306If you are calling one of those function, it will cause the program to shut down immediately. Internally log.Fatal(ln|f) will call os.Exit(1). Everything will be stopped, and the deferred functions will not be called. It’s a sort of emergency button. The program does not terminate properly when you do a fatal.
will output to the standard logger the return value of fmt.Sprint and call os.Exit(1)
the return value of fmt.Sprintf will be printed to the standard logger and os.Exit(1)will be called
the return value of fmt.Sprintln will be printed to the standard logger and os.Exit(1) will be called
The same logic as above is used for calls to log.Panic[f][ln] there is no difference, fmt.Sprint , fmt.Sprinf and fmt.Sprintln is called under the hood.
The only difference is that we have a call to panic (instead of an os.Exit(1)).
When the logger is calling panic on your behalf :
Any deferred functions defined in the current function will be called
The current function will return to its caller
In the caller, all deferred functions defined will be called, and the caller will return to the function that called it...
A call to panic will call all deferred functions that are defined in the goroutine call stack.
The functions defined on the standard logger are limited, and you might want to add fancy features to your application’s logger. You can use a popular module called logrus (more than 17k stars on Github)1.
\n\nFirst, you need to install it (after initializing your module with go mod init your/module/path) :
go get github.com/sirupsen/logrusThen you can configure your logger :
\nfile, err := os.OpenFile("app.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)\nif err != nil {\n // handle error\n}\nlog.SetOutput(file)Here we will log to a file. You can also log to the standard error (stderr). In this case, you can omit the last lines.
\nThen you can begin to log :
\nlog.Print("my first log")Here is the output :
\ntime="2021-02-19T12:28:57+01:00" level=info msg="my first log"An interesting feature of logrus is to handle log levels. Each log has a level of criticality. With a level, you can apply filters to your log files. Here is an example of a log for each level available :
\n// logging/main.go \n\nlog.Trace("going to mars")\nlog.Debug("connected, received buffer from worker")\nlog.Info("connection successful to db")\nlog.Warn("something went wrong with x")\nlog.Error("an error occurred in worker x")\nlog.Fatal("impossible to continue exec")\nlog.Panic("system emergency shutdown")trace: very detailed information about the program run, usually for debugging purposes
debug: used to output very detailed information, usually for debugging purpose
info: informational logs
warn (warning): a non-critical error or event that we should fix. The application can still operate.
error: an error that we should fix. The application can still operate.
fatal: a critical error, the application will be stopped
panic: the highest level of criticality, the application will be stopped
You can then set at which level your application will produce logs.
\nBy default, trace and debug levels are not printed.
\nFor example, if you want to output all levels use the following line :
\nlog.SetLevel(log.TraceLevel)If you want to output only errors (and also fatal / panic) :
\nlog.SetLevel(log.ErrorLevel)You can add fields to your log messages. A field consists of a key and a value. A monitoring solution can easily parse fields
\n// logging/main.go \n\nworkerLogger := log.WithFields(log.Fields{\n "source": "worker",\n})\nworkerLogger.Info("worker has finished processed task")\n\nmysqlLogger := log.WithFields(log.Fields{\n "source": "db",\n "dbType": "mysql",\n})\nmysqlLogger.Error("impossible to establish connexion")Will generate the following logs :
\ntime="2021-02-19T12:49:47+01:00" level=info msg="worker has finished processed task" source=worker\ntime="2021-02-19T12:49:47+01:00" level=error msg="impossible to establish connexion" dbType=mysql source=dbSyslog is an open standard for message logging. This section will detail some best practices extracted from RFC 5424, which specifies the format of log messages. Here is a list of fields that you can add to your logger to increase its efficiency :
\nit describes which part of the system has sent the log message
\nThe RFC gives the set of all possible facilities:
\n| Code | \nFacility | \n
|---|---|
| 0 | \nkernel messages | \n
| 1 | \nuser-level messages | \n
| 2 | \nmail system | \n
| 3 | \nsystem daemons | \n
| 4 | \nsecurity/authorization messages | \n
| 5 | \nmessages generated internally by syslogd | \n
| 6 | \nline printer subsystem | \n
| 7 | \nnetwork news subsystem | \n
| 8 | \nUUCP subsystem | \n
| 9 | \nclock daemon | \n
| 10 | \nsecurity/authorization messages | \n
| 11 | \nFTP daemon | \n
| 12 | \nNTP subsystem | \n
| 13 | \nlog audit | \n
| 14 | \nlog alert | \n
| 15 | \nclock daemon (note 2) | \n
| 16 | \nlocal use 0 | \n
| 17 | \nlocal use 1 | \n
| 18 | \nlocal use 2 | \n
| 19 | \nlocal use 3 | \n
| 20 | \nlocal use 4 | \n
| 21 | \nlocal use 5 | \n
| 22 | \nlocal use 6 | \n
| 23 | \nlocal use 7 | \n
The Go Syslog package has already an enum of the facilities, so you do not have to redevelop it :
\n//src/log/syslog/syslog.go\n//...\nconst (\n // Facility.\n\n // From /usr/include/sys/syslog.h.\n // These are the same up to LOG_FTP on Linux, BSD, and OS X.\n LOG_KERN Priority = iota << 3\n LOG_USER\n LOG_MAIL\n LOG_DAEMON\n LOG_AUTH\n LOG_SYSLOG\n LOG_LPR\n LOG_NEWS\n LOG_UUCP\n LOG_CRON\n LOG_AUTHPRIV\n LOG_FTP\n _ // unused\n _ // unused\n _ // unused\n _ // unused\n LOG_LOCAL0\n LOG_LOCAL1\n LOG_LOCAL2\n LOG_LOCAL3\n LOG_LOCAL4\n LOG_LOCAL5\n LOG_LOCAL6\n LOG_LOCAL7\n)\n//...each log message do not have the same seriousness; some messages are pure information, but others require you to take immediate action to maintain the service up. For instance, if you are running out of disk space, the log should be at a high level of severity because if you do not take action immediately, it will make your service crash.
\nThe specification gives us also a list of severity level that is interesting :
\n| Code | \nLevel | \n
|---|---|
| 0 | \nEmergency: the system is unusable | \n
| 1 | \nAlert: we must take action immediately | \n
| 2 | \nCritical: critical conditions | \n
| 3 | \nError: error conditions | \n
| 4 | \nWarning: warning conditions | \n
| 5 | \nNotice: normal but significant condition | \n
| 6 | \nInformational: informational messages | \n
| 7 | \nDebug: debug-level messages | \n
The go Syslog package also has an enum for the severity :
\n//src/log/syslog/syslog.go\n//...\nconst (\n // Severity.\n\n // From /usr/include/sys/syslog.h.\n // These are the same on Linux, BSD, and OS X.\n LOG_EMERG Priority = iota\n LOG_ALERT\n LOG_CRIT\n LOG_ERR\n LOG_WARNING\n LOG_NOTICE\n LOG_INFO\n LOG_DEBUG\n)\n// ...this is the date, time, and timezone at which the log has been written (and in extension when the event has happened). The RFC 5424 recommends using a specific format (specified in the RFC 3339). To get the timestamp in the requested format, you can use the following code :
\ntimestamp := time.Now().Format(time.RFC3339)when you aggregate logs, they can come from different machines; you have to be able to know which machine has encountered a problem an easy way to get it is to make a call to os.Hostname(). This last instruction will ask the kernel to answer the question :
hostname := os.Hostname()populate this property with the name of your program. This will allow you to differentiate in the logs which application has produced the logs. It can be a time-saving habit if you have several applications that run on the same machine. Example: router
\nif you want to make your logs usable, you have to make them parsable. To do so, you can add structured data to your log. By structures data, we mean a set of key-value pairs that brings context and additional information to your log.
\nThe priority value (named PRIVAL) is equal to facility * 8 + severity
<9>1 2018-10-17T20:56:58+02:00 Mx.local router help me pleaseWe can send logs to a log deamon. A deamon is a program that runs continuously in the background. A deamon log is a program that is responsible for collecting logs from applications :
\nlogwriter, err := syslog.New(syslog.LOG_WARNING|syslog.LOG_DAEMON, "loggingTestProgram")\nif err != nil {\n log.Fatal(err)\n}\nlogwriter.Emerg("emergency sent to syslog. TEST2")We will add the log to the system logs of the machine. Because it’s an emergency, it will be broadcasted to open terminals (see figure).
\nA call to log.Fatal will execute deferred functions. True or False?
A call to log.Panic will execute deferred functions. True or False?
Order the following log levels in terms of criticality (from low to high): error, panic, info, trace.
Why adding structured data to your logs is considered a good practice?
A call to log.Fatal will execute deferred functions. True or False?
False
Deferred functions will not be run
The program will exit with code 1
A call to log.Panic will execute deferred functions. True or False?
Order the following log levels in term of criticality (from low to high): fatal, error, panic, info, trace
\nWhy adding structured data to your logs is considered a good practice?
\nStructured data are easy to parse by monitoring systems
Consequently, logs with structured data can be exploited more efficiently by monitoring tools and teams.
An application should produce informative logs
Logs can help developers to debug their applications when a failure occurs.
We can write logs on a file or on any type that implements the interfaceio.Writer
The log package from the standard library is a good starting point when you need to add logs to your application.
When log.Fatal is called deferred functions are not run
When log.Panic is called deferred functions are run
The logrus module offers interesting features such as log levels, formatting options, and fields
Adding parsable fields to your logs is a good practice; it makes them easier to analyze.
https://github.com/sirupsen/logrus↩︎
Previous
\n\t\t\t\t\t\t\t\t\tConcurrency
\n\t\t\t\t\t\t\t\tNext
\n\t\t\t\t\t\t\t\t\tTemplates
\n\t\t\t\t\t\t\t\t