Terminologia

Slide 1 - Foundations - Definitions terminology

Esta sessão é o trabalho de base e faz parte da secção de Terminologia. Iremos introduzi-lo sobre a linguagem, terminologia e definições de alguns dos conceitos básicos, funções e processos que vai colocar em uso durante o resto do curso.

Image by Carsten Witzel, obtained via http://www.freeimages.com/photo/root-of-a-tree-1371557.

Image by Carsten Witzel, obtained via http://www.freeimages.com/photo/root-of-a-tree-1371557.

Slide 2 - Key concepts

Primeiro, vamos rever a estrutura desta sessão.

Na secção de DEFINIÇÕES primeiros tentaremos esclarecer a utilização da palavra "Dados".

A seguir, veremos a terminologia em torno de Campos, Etiquetas e Listas.

E por fim, vamos esclarecer alguns conceitos sobre caracteres e seus mapeamentos que ajudarão a evitar confusões de dados indesejadas.

Slide 3 - Definitions

Primeiras definições.

Nesta secção analisaremos alguns dos muitos termos e frases diferentes que existem no mundo da gestão dos dados e da informação.

Vamos utilizar esses recursos ao longo do curso e é importante que você esteja familiarizado com quando e como usá-los.

Image by Sharon Grant: https://www.flickr.com/photos/rondls_pics/3629532537/in/album-72157619834323354/

Slide 4 - Definitions - a rose by any other name

Vamos focar por um momento em algumas palavras e frases comuns que contenham a palavra dados.

Slide 5 - What’s the difference - Database

Todas as frases aqui incluem a palavra dados, mas são todas iguais, e, se não, então qual é a diferença?

Slide 6 - Database

Então, o que é uma "base de dados"?

Slide 7 - Database language

Vamos dar uma olhada na frase "linguagem da base de dados".

According to the Oxford English dictionary “language” is the method of human communication…​consisting of the use of words (or symbols) in a structured and conventional way.

Para nossos propósitos, as linguagens de base de dados e linguagens aliadas mais próximas são a forma pela qual um ser humano se comunica com um computador para manipular e circular em torno de dados.

Os exemplos que você pode ter encontrado são:

Slide 8 - Database program/software

Agora vamos dar uma olhada na frase "programa de base de dados ou software".

Mais uma vez, de acordo com o dicionário inglês Oxford, um "programa" é um conjunto de medidas ou actividades relacionadas com uma finalidade de longo prazo.

Neste contexto, um "programa de base de dados ou software" combina e apresenta funções e recursos para a manipulação de dados, sendo uma interface unificada.

Os exemplos que você pode encontrar são:

Acesso Microsoft, FilemakerPro e Oracle

Portanto, programas ou softwares de bases de dados são utilitários agrupados que permitem o utilizador, através de uma linguagem de base de dados, armazenar, gerir, consultar e reportar dados presentes no sistema.

Slide 9 - Data cleaning tool

O último termo é "ferramenta de limpeza de dados".

Data cleaning tools are systems whose utilities focus is on the manipulation of data with the aim of increasing fitness for use.

Os exemplos que você pode estar familiarizado ou aprenderá durante o curso são:

OpenRefine, MatLab and spreadsheets

A ferramenta de limpeza de dados deve ajudar o utilizador na descoberta e reparação de dados confusos para aumentar a qualidade.

Slide 10 - Definitions - a rose by any other name

Continuando com mais alguns termos que você deve estar familiarizado, veremos as diferenças entre os termos Campo, Etiqueta e Lista.

Slide 11 - Field vs label

You will often hear reference made to field or column names and field labels or headings. Often they will be used interchangeably, however they are different things.

Field or column names refer to the actual column in the database table. They are a often short, not always in plain language, sometimes computer generated; but in a properly formed database they are ALWAYS unique. Field names are almost always used by database languages, because the avoid ambiguity.

Labels and headers on the other hand, refer to the human readable versions of the field names. Most often you will see these in the graphical user interface of your software and on exports and reports. In some software you can in fact have different labels to the same field on a user by user or group basis!

Slide 12 - Lists

As you work with your data you will come across the phrases “drop-down list” and “controlled vocabulary” and on the surface they can appear to be the same. Indeed they are both powerful tools to help you keep your data clean, but they are different.

A drop-down list, also sometimes called a lookup table is a data entry element that is created in your database software. It allows to you select a value from a predefined set of options. They can also be generated from the data itself on the fly.

A controlled vocabulary on the other hand, is a very strictly defined, domain specific set of values that should be applied to a particular field. They are an integral part of a well defined standard. Such values can be used in a drop-down list as mentioned previously.

Slide 13 - Definitions - a rose by any other name

Nesta secção, veremos o que o termo significa em relação aos dados e computadores.

Slide 14 - Characters

Como lembretes os computadores "vêem" em binário. Tudo o que você digita é mantido na memória como uma série de uns e zeros.

The computer does not “see” the repertoire of letters and symbols on your keyboard. It needs a map to translate the character set that you would write or type, into its binary language and then back again to show it to you on the screen or a printer.

Este mapa é composto por duas partes:

The “coded character set”

e

A "codificação de caracteres"

A cada símbolo ou glyph é atribuído um valor numérico e, em seguida, esse número é convertido em binário.

Slide 15 - Characters example

Neste exemplo, vemos como a codificação de caracteres é usada para mapear a letra A no alfabeto ocidental.

O valor ASCII para a letra maiúscula A é o número decimal 65

que em HEXIDECIMAL é 041

e em BINÁRIO 01000001

Tente mapear seu próprio nome para Binário usando o link na parte inferior da tela.

Slide 16 - Characters comparison

In effect what is happening is that when you type into a keyboard and save a file the software that you are using will select an encoding to save with it.

Quando você abrir novamente esse ficheiro, o software precisa saber qual codificação (mapa) deve ser usada para exibir novamente as informações de volta para você, correctamente.

Slide 17 - Characters

Então, por que isso é importante para você como detentor de dados?

Digamos que você traduza o início da obra de Hamlet

 "Ser ou não ser, eis a questão"

em Francês e depois salvar o ficheiro com um editor de texto que usa a codificação Latin-1

Depois envia para um colega para verificar a tradução. Voila!

Slide 18 - Characters

Quando eles abrem o arquivo em Microsoft Word é isso que eles veem.

Mon dieu!

Há alguns caracteres inesperados.

Esqueceu-se de dizer ao seu colega que está a usar Latin-1 como codificação.

O Microsoft Word no seu computador foi definido para usar a codificação Unicode UTF-8. Os mapas não correspondem.

Dependendo do software, os dados podem estar irremediavelmente corrompidos.

Slide 19 - Review

Em revisão, analisamos as diferenças entre as frases:

E mostrou que embora os termos possam parecer semelhantes, na verdade têm definições muito diferentes.

Slide 20 - Wrap-up

Lembre-se que:

“…​The first step in wisdom is to know the things themselves; this notion consists in having a true idea of the objects.”

We often take for granted that there is a whole language associated with biodiversity informatics that is not commonplace and has to be learned. There is no shame in going back to basics and getting to grips with the vocabulary, grammar and syntax just like with any new language.

Slide 21 - Conclusion

This video is part of a series of presentations used in the GBIF Biodiversity Data Mobilization course. The biodiversity data mobilization curriculum was originally developed as part of the Biodiversity Information Development Programme funded by the European Union.

Esta apresentação foi originalmente criada e narrada por Sharon Grant com contribuições adicionais Treinadores BID e BIFA, Mentores e Alunos.

Slide 1 - Foundations - Software terminology

This session is foundation work and part of the Terminology section We will be introducing you to types of software used in data mobilization.

Image by Carsten Witzel, obtained via http://www.freeimages.com/photo/root-of-a-tree-1371557.

Slide 2 - Software for mobilization

The purpose of this short presentation is to give you some examples of the different types of applications and software available in the world of biodiversity mobilization informatics.

Hopefully it will help you to start to think about what uses they may be put too and what types of things might be practical for you.

Slide 3 - Software definition

So very briefly, a definition.

We are making a distinction here between all software and those applications which have been created for an end-user to carry out a task.

We will take a look at software that can be used for:

Slide 4 - Software for data capture

In the realm of data capture software, there is often no need to go “big” when capturing information initially, especially in the field.

You can capture information from collection specimens or live observations in a simple spreadsheet such as:

Excel, OpenOffice, LibreOffice or Google Sheets.

There may even not be a need to create your own system at all. Apps such as iNaturalist, or eBird are stable, robust and have simple intuitive interfaces.

Whichever the choice, it is important to think about the long term preservation of the data that you collect.

Slide 5 - Software for data management

So, depending on your needs, a more complex data management software might be the right solution for your data. Especially if long term retention and preservation is a priority.

More stable management platforms also better support sharing of data to multiple audiences.

There are many choices which vary in terms of cost and support and a few are shown here. From left to right:

Symbiota, Specify, EMu, Elysia and Brahms.

Slide 6 - Software for data capture and management

If a home-grown solution is preferred and can be supported, relational database software can provide cheaper alternatives.

Examples incluse MySQL, SQLServer, Access or filemaker

Slide 7 - Software for data cleaning

Increasingly the lines between languages, programs and tools have blurred, each being capable of many complex and overlapping functions. The common “Find and Replace” function is a good example of a data cleaning tool that is found in almost all database programs and is a command in database languages.

You are probably already familiar with spreadsheets such as Excel and indeed these are probably the most common data cleaning tools available. However there are others.

OpenRefine originally known as GoogleRefine is a powerful free, open source data cleaning tool with a large support community. You will see more of this program in upcoming sessions.

R is another increasing used, sophisticated programming language and environment. With a little practice it can also provide powerful data manipulations.

Slide 8 - Software for data publishing

Planning further ahead to the sharing of your data through GBIF to the international scientific and deciders community you will need to choose a publishing tool.

The choice will depend largely on the level of infrastructure support at your institution. If this is high and stable, installing an IPT or BioCASe application may be the right option. But there are others if this is not the case.

You can consider partnering with an existing ipt provider or selecting a database management system that includes a publishing pathway of its own.

Whichever you choose, make sure that your system allows you to export data as csv or txt files and you will ultimately be able to publish your data.

Slide 9 - Other software to handle data

This is a summary of the information collected in the course survey of software in use by students, mentors and trainers in the GBIF community.

The link and reference to more details of each section, is at the bottom of the slide.

Reference: https://docs.google.com/spreadsheets/d/1IWBsL4r-LsJRdYYPy9q4eLOAa1nzZYBwTdwk3Kd8cV8/

Slide 10 - Wrap-up

Lembre-se que:

“The purpose of software is to help people” and not the other way around.

Your choice of solution should make your life easier but it should also not sacrifice the long term viability of the data that it is meant to work with.

Slide 11 - Conclusion

This video is part of a series of presentations used in the GBIF Biodiversity Data Mobilization course. The biodiversity data mobilization curriculum was originally developed as part of the Biodiversity Information Development Programme funded by the European Union.

This presentation was narrated by Sharon Grant with additional contributions by BID and BIFA Trainers, Mentors and Students.

Slide 1 - Foundations - Structures terminology

Esta sessão é o trabalho de base e faz parte da secção de Terminologia. Iremos introduzi-lo sobre a linguagem, terminologia e definições de alguns dos conceitos básicos, funções e processos que vai colocar em uso durante o resto do curso.

Image by Carsten Witzel, obtained via http://www.freeimages.com/photo/root-of-a-tree-1371557.

Slide 2 - Key concepts

Primeiro, vamos rever a estrutura desta sessão.

In the section on STRUCTURES we will be working on understanding the field and data types that hold data, the structures that help to organize and protect that data and lastly look at what these mean for the integrity and security of your data.

Slide 3 - Structures

In this section on basic database Structures we will focus on the different elements that databases, software tools and languages use to handle the data that you enter.

Slide 4 - Structures - Planes, trains and automobiles

First, lets look at field types, also known as data types.

Slide 5 - Containers – field/data types

Data Type lets the computer know what kind of data is expected in a particular field. This then dictates what functions can be performed on it.

They can be simple and encompass a single concept or combine concepts depending on the software or language you are using. Either way the essential building blocks are the same.

So let’s look at six that you will come across most frequently.

Slide 6 - Containers – field/data types

First Numeric fields – This data type holds numbers, and there are 2 kinds:

Integers and long integers, hold whole numbers. For example 1, 2, 3, 99, 2000.

The size of the number, depends on the number of bits assigned to the field (more about that later). For now just remember that a long integer field will hold a bigger number than an integer.

Floats, also referred to as doubles, hold fractions or parts of whole numbers. For example 4.5, 19.47

The important thing to remember here is that a float always has decimal places so the number 10 will be stored as 10.0. The number of decimal places being an integral element of the type. This has implications when it comes to precision and accuracy.

Slide 7 - Containers – field/data types

Alphanumeric fields - are also referred to as text, character or string fields.

They can hold any letter, number or symbol in the coded character set and usually have a defined character limit. Entering more characters than the character limit can often result in truncation and loss of data on save.

Slide 8 - Containers – field/data types

Another type of text field that you will come across is the Unstructured text field, also known as memo, long char or blob fields.

These types of field are used to hold large amounts of unformatted text and unlike alphanumeric fields may not have an obvious character limit.

In simple systems, they are generally less flexible in terms of processing; for example sorting and indexing, than their smaller alphanumeric cousins.

With decreased storage costs and the rise of NoSQL document-oriented database programs, such as MongoDB however, this is changing.

Slide 9 - Containers – field/data types

System Value fields have a double nature and what you see on the screen is not usually the actual value that the software is storing in memory. Date and time fields are the most common example of this type of field.

In this example you typed “1 March 1971” into a cell in Excel and hit return.

Now it says “1-Mar-71” in the cell AND “3/1/1971” in the functionbar.

Excel is applying a display format to the cell, which you can change at will.

HOWEVER, if you remove the format you see that what is actually being stored in the field is the number 25993.00!

So you can see that the field is actually a float. You should beware because that system value not be the same from software to software. Exports should to checked to make sure the actual date is exported and not the system value.

Slide 10 - Containers – field/data types

Boolean fields – also known as Binary fields only contain one of two values that represent a one or a zero.

They are a special case of a System Value field as mentioned previously in that whilst the software actually saves a number it often displays values such as:

Yes/No; Yes/no; Y/N; True/False.

You should take care to find out which way around the system you are using translates.

Is a 1 a yes or no?

Slide 11 - Containers – field/data types

Structured Text fields are less common but mentioned here for completeness.

They are fields which can be either made up of combinations of the simple data types or even be defined by the data itself.

Most often these are used in programming languages to allow flexibility and handling of complex concepts.

An example is the factor field used in the R programming language, which is defined as a type vector containing a set of numeric codes with character-valued levels.

Ref:  https://www.stat.berkeley.edu/~nolan/stat133/Fall05/lectures/DataTypes4.pdf

Slide 12 - Structures - planes, trains and automobiles

Now that you have an understanding of the field types used to represent your data elements let’s talk about the containers or structures, that many database programs and software use collate and display your data.

Reference: https://www.geeksforgeeks.org/data-structures/

Slide 13 - Containers - Data structures

First is the cell.

This is the smallest data structure, it holds a single value and has a single data type associated with it.

Slide 14 - Data structures

There is one concept that you should beware of when it comes to cells and that is the “empty cell”.   “An empty cell doesn’t necessarily mean an empty field.”

Some software “fills” an empty field with a System value based on the data type.

Examples are “NULL” “N A” or even zero!

Slide 15 - Containers - Data structures

As you start to organize cells you begin to build a grid. Grids have implied structure in the form of

Rows

Slide 16 - Containers - Data structures

E colunas.

Slide 17 - Containers - Data structures

When you organize cells into rows and columns, obviously, what you get is a grid!

This is looks very much like a spreadsheet right?

Well, yes. The difference however between a spreadsheet grid and a table are the strength of the ties that hold the rows and columns together.

Slide 18 - Structures - planes, trains and automobiles

So let’s now talk about those connections.

What the implications for the integrity and security of your data of loose unenforced relationships between rows and columns?

Slide 19 - Data structures

What actually makes a table useful are the connections that we create to give the data in the structure meaning.

We intrinsically make patterns of attributes that turns data into information.

Rows become records and Columns become attributes

In a spreadsheet however, the connections between cells are idiomatic, loose and un-enforced.

It is easy for the attributes between the rows to get jumbled up or for the values get changed and then the meaning is lost.

Slide 20 - Integrity and security

This lack of enforced connections makes a spreadsheet easy to use but these same qualities also cause problems.

Errors are easy to make and hard to catch.

Connections and relationships are not documentable and so subject to interpretation and mutation.

Spreadsheets are not built to scale either in terms of size or numbers of users.

So can be slow, crashy and easily corruptible

Spreadsheets are easy to copy and email

So they are hard to version and keep track of

The question is what is the solution?

Reference: https://www.mrc-productivity.com/blog/2014/09/7-dangers-of-spreadsheets-part-1/

Slide 21 - Containers - Integrity and security

Let’s look back, at the connections in a bit more detail.

We generally understand that the data in each row of a table means something.

For example each row can represent a specimen or a taxa that was found at a particular place, at a particular time, by a particular person.

It would be odd to have one row in a table that represents a specimen followed by a row that represents a collection site.

We also often want to analyse, sort and compare values of particular attributes between rows.

For example you may want find the mean weight of all the Kenyan bats.

So ideally we want to use a system to store our data that:

These are functions of a database!

Slide 22 - Integrity and security

Functions in a true database inherently bind rows together and columns may only have a single data type.

Tables in database are held together by virtual links called keys.

These keys are identifiers that match between tables and allow data to be connected efficiently.

It is these 2 key concepts that distinguish a spreadsheet from a database.

Reference: https://www.mrc-productivity.com/blog/2014/09/7-dangers-of-spreadsheets-part-1/

Slide 23 - Review

In review then, we have looked at the 6 most common data types, the basic building blocks of data structure and distinguished between a spreadsheet grid and database table.

Understanding these elements and how they interact with one another will allow you to better determine the best ways to manipulate and document the data that falls under your custodianship.

Slide 24 - Wrap-up

Remember that:

“When you realize the difference between the container and the content, you will have knowledge.”

Using the correct terms to describe both your data and the structures that you create and use to store it; is the most important way to ensure that it remains relevant and useable.

Never underestimate how much damage can be done to priceless information by careless descriptions and storage.

Slide 25 - Conclusion

This video is part of a series of presentations used in the GBIF Biodiversity Data Mobilization course. The biodiversity data mobilization curriculum was originally developed as part of the Biodiversity Information Development Programme funded by the European Union.

Esta apresentação foi originalmente criada e narrada por Sharon Grant com contribuições adicionais Treinadores BID e BIFA, Mentores e Alunos.