LISTAGG / implode solution for SQL Server

Group multiple records into one and concatenate the values of the string/text values of a field:

ALTER view [dbo].[vcomments] as
with x as
(
select
cmt_domain,
cmt_indx,
cmt_seq,
concat(cmt_cmmt1,cmt_cmmt2,cmt_cmmt3,cmt_cmmt4,cmt_cmmt5,cmt_cmmt6,cmt_cmmt7,cmt_cmmt8,cmt_cmmt9,cmt_cmmt10,cmt_cmmt11,cmt_cmmt12,cmt_cmmt13,cmt_cmmt14,cmt_cmmt15) as cmt_cmmt,
cmt_lang
from
v_cmt_det_act
)
select
cmt_domain,
cmt_indx,
--FOR XML PATH returns a one record XML datatype in which all records are "unioned".
-- However, special characters are encoded as excape sequences, i.e. "<" becomes "<"
-- To reverse the encoding, get the entire root document/element as a value using the VALUE function.
-- The [1] is required because VALUE expects a singleton, i.e. exactly ONE element
rtrim(substring(
(
select char(10) + '[PAGE ' + cast(x2.cmt_seq as nvarchar(2)) + ']' + char(10) + x2.cmt_cmmt -- a field with no name gets no XML element to enclosed it. In short, the XML tag is removed
from x x2
where x1.cmt_domain = x2.cmt_domain and x1.cmt_indx = x2.cmt_indx
order by x2.cmt_seq
for XML PATH (''), type -- the brackets indicate the XML root element. Specifying an empty string as root element name removes it
).value('(/)[1]','nvarchar(max)')
,2,100000)) as cmt_cmmt
from
(
select distinct cmt_domain, cmt_indx
from x
) x1

Query Active Directory from SQL Server

Get active directory server and LDAP servers in a domain
nltest /dclist:sub.dom.com

results amongst others in:
\\dcserver1.sub.dom.com -> dcserver1 is the AD server

Analyse active directory structure
Use Sysinternals ADExplorer to analyse the structure of active directory. Connect to server: dcserver1

Users in a group from command prompt
net group adtestgroup /domain

Active Directory structure in FROM clause
E.g. FROM "LDAP://sub.dom.com/OU=Groups,OU=Global,DC=sub,DC=dom,DC=com"

LDAP = case-sensitive protocol name, always this value
Before the slash: sub.dom.com, is the domain for which to find the AD structure
After the slash: the part of the tree to search. Specified from right to left. So in the example, from parent to child:
com -> dom -> sub -> Global -> Groups -> ADTestGroup, searches the nodes beneath the lowest level, which is ADTestGroup

The names ("dom", "Groups", etc), type ("OU", "CN", "DC") and depth of the levels are dynamic. So check with ADExplorer how AD is configured in the specific instance.

Users in a group from SQL Server
First, add a linked server to ADSI (fill with correct password for DOMAIN\ACCOUNT):

exec sp_addlinkedserver @server = 'ADSI', @srvproduct = 'Active Directory Services 2.5', @provider = 'ADSDSOObject', @datasrc = 'adsdatasource'
exec sp_addlinkedsrvlogin @rmtsrvname = 'ADSI', @useself = 'False', @locallogin = null, @rmtuser = 'DOMAIN\ACCOUNT', @rmtpassword = '********'

Select users in group "adtestgroup":

select * from openquery(ADSI,'
    SELECT objectCategory, cn, sn, mail, name, department, company
    FROM ''LDAP://sub.dom.com/DC=sub,DC=dom,DC=com''
    WHERE MemberOf=''CN=adtestgroup,OU=Groups,OU=Global,DC=sub,DC=dom,DC=com''
    ORDER BY cn
')

Unfortunately, dynamic sql is not possible inside a udf, so a procedure is needed:

alter procedure cst_usersingroup(@grp nvarchar(100))
as
begin
    declare @sql as nvarchar(max) = '
    select samaccountname as accountname, name, mail, department, title, company
    from openquery(ADSI,''
        SELECT objectCategory, cn, sn, mail, name, department, company, title, samaccountname
        FROM ''''LDAP://sub.dom.com/DC=sub,DC=dom,DC=com''''
        WHERE MemberOf=''''CN=' + @grp + ',OU=Groups,OU=Global,DC=sub,DC=dom,DC=com'''''')
    x
    order by samaccountname'

    exec(@sql)
end

Select all BI_* and EDW_* groups from active directory:

select lower(cn) as grp, whencreated, whenchanged, distinguishedname
from openquery(ADSI,'
    SELECT cn, distinguishedname, whencreated, whenchanged
    FROM ''LDAP://sub.dom.com/OU=Groups,OU=Global,DC=sub,DC=dom,DC=com''
    where CN=''bi_*'' or  CN=''edw_*''
') x
order by cn

Notes:

• In most cases, all domain users can query AD using the basic search method specified before. More advanced AD search methods might be disabled and require special rights

Create schema with separate autorization

Perform the following steps in SQL Server to create a schema called "bi" and give create view rights to user "domain\testuser":

CREATE SCHEMA [bi]
CREATE ROLE [db_bischema]
ALTER AUTHORIZATION ON SCHEMA::bi TO [db_bischema]
GRANT CREATE VIEW TO [db_bischema] -- "create view" includes the power to drop views
ALTER ROLE [db_bischema] ADD MEMBER [domain\testuser]

Advantages:

• If views are removed from or added to the schema, autorization does not need to be added to each individual view
• Give certain users the rights to (re)create the views inside the schema, without the risk of modification of the rest of the database

Quipu JDBC connection to MS Access

There are several ways to connect to a MS Access database from Java, like the low-level MS Access library called Jackcess: http://jackcess.sourceforge.net/

The downside is that most Java applications that support heterogeneous connections have a default built-in procedure to connect and open the database. Most of the time, the "getSchemanames" function is called and this one doesn't function as expected with MS Access. Quipu uses this method also.

Therefore, a connection can be made in the following way, but results in an error when retrieving the schema names during reverse engineering with Quipu from MS Access

1) JDBC to ODBC bridge

Create a connection to the MS Access database d:\test.mdb using a native and DSN-less JDBC to ODBC bridge:

Manage connection types:
Add a new connection type:

name= ms access jdbc
type= jdbc
database or file delimiter=
database or file required= false
jdbc port delimiter= :
jdbc default port=
jdbc driver= sun.jdbc.odbc.JdbcOdbcDriver
url type= jdbc:odbc:Driver={Microsoft Access Driver (*.mdb, *.accdb)};DBQ=

Manage connections:
Add a new connection:

name: ms access jdbc
database= test.mdb
hostname= d
port= /

Notes:

• The url type above is for Java x64 and MS Access x64. If Java x32 is used, then the url type= jdbc:odbc:Driver={Microsoft Access Driver (*.mdb)};DBQ=
• No additional jdbc driver (jar file) needs to be installed, because the driver is embedded in the Java JRE runtime

There is an alternative method:

2) UCanAccess

It is a open source database driver written by Marco Amadei that behaves identical to the JDBC to ODBC bridge, but
- doesn't use ODBC
- handles the getschemanames call, using a default "PUBLIC" schema

Download it here: http://sourceforge.net/projects/ucanaccess

To use it with Quipu:
- Extract the jar files from the \lib folder in the zip file to Quipu's \webapps\qp\WEB-INF\lib folder
- Sort files by name and make sure no duplicates exist, e.g. hsqldb and hsqldb-1.8.0 is not allowed.
- Remove the duplicate with the lowest versionnumber. In the case of hsqldb, which has no versionnumber: keep the one from the UCanAccess zipfile

Manage connection types:

Add a new connection type:
name= ucanaccess
type= jdbc
database or file delimiter=
database or file required= false
jdbc port delimiter= :
jdbc default port=
jdbc driver= net.ucanaccess.jdbc.UcanaccessDriver
url type= jdbc:ucanaccess://

Manage connections:

Add a new connection:
name: ucanaccess
database= test.mdb;showschema=true
hostname= d
port= /

Notes:

- The "showschema=true" property bypasses the getSchemanames error

Reference SQL Server table with spaces from Oracle

It is possible to select data from a SQL Server table that contains spaces by using a database link from Oracle.

The syntax is as follows:

SELECT * FROM "DBO$TABLE WITH SPACES"@MYDBLINK

with

SQL Server login/user: dbo
SQL Server table name: table with spaces
DB link: mydblink

Query Oracle from SQL Server

Start with the following steps:

• Install the Oracle Instant Client (OIC) on the machine where SQL Server is running and add the "Oracle Provider for OLE DB" driver, e.g. to folder c:\oracle11g.
• For a SQL Server x64 version running on a Windows x64 OS download the instantclient-basiclite-windows.x64-11.2.0.2.0.zip. This is a 22 MB download.
• Then add the Oracle Data Access components, to get the "OLE DB for Oracle" driver:  http://www.oracle.com/technetwork/database/windows/downloads/index-090165.html
  At the time of writing (March 2012): select the "64-bit ODAC 11.2 Release 4 (11.2.0.3.0) Xcopy for Windows x64". This is a 54 MB download
• Inside SQL Server Management Studio:
  Linked Servers -> Providers -> OraOLEDB.Oracle -> Allow inprocess = true
• Optional: to enable adhoc querying (querying without a linked server), then enable this:
• SQL Server 2005 via surface area configuration tool
• SQL Server 2008 via facets: inside SQL Server Management Studio, right click the server -> facets -> facet: surface area configuration -> AdHocRemoteQueries = true
• Optional: if the connection to Oracle should be based on a TNS names entry, then add an "admin" folder to the OIC folder. Put or create an tnsnames.ora file there, e.g. c:\oracle11g\admin\tnsnames.ora
• Optional: use "tnsping" to test the entry from the command prompt, e.g. "tnsping dwh.world". If it doesn't recognize the entry, then create an environment variable called "TNS_ADMIN" and let it point to this "admin" folder, e.g. c:\oracle11g\admin
• Optional: if the following steps to get data from an Oracle instance don't work, then edit this registry key: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSDTC\MTxOCI.
  Set the following values:
  OracleXaLib=oraclient11.dll
  OracleSqlLib=orasql11.dll
  OracleOciLib=oci.dll

Note: these DDLs can be found in the c:\oracle11g\product\11.2.0\client_1\BIN folder. The specific version depends on the Oracle Client and might be for example oraclient10.dll en orasql10.dll. Use the names of the DDLs as found in this folder.

Three ways to get the data from an Oracle instance

Example:

Server: locksnlatches
Oracle instance (SID): dwh
Port: 1521
Schema/user: scott
Password: tiger
tnsnames.ora entry:

dwh.world =
(DESCRIPTION=
  (CID=DWH_APP)
  (ADDRESS_LIST=
    (ADDRESS=
      (PROTOCOL=TCP)(HOST=locksnlatches)(PORT=1521)
     )
   )
  (CONNECT_DATA=
    (SID=dwh)
  )
)

1) Using a linked server

• Linked Servers -> New linked server... ->  General ->
• linked server: lnk_locksnlatches (this name can be anything)
• provider: Oracle provider for OLE DB
• product name: oracle (this name can be anything)
• data source: (DESCRIPTION=(CID=DWH_APP)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=locksnlatches)(PORT=1521)))(CONNECT_DATA=(SID=dwh)))
• provider string: leave empty!
• location: leave empty!
• catalog: leave empty!
• Linked Servers -> New linked server... ->  Security ->
• Be made using this security context
• Remote login: scott
• With password: tiger

Use the four-point notation to retrieve data from Oracle:

SELECT 1 FROM lnk_locksnlatches..scott.dual

Note: the "security" tab specifies the Oracle schema/user and password used by the connection!

2) Without a linked server (TNS-less)

SELECT * FROM OPENROWSET('OraOLEDB.Oracle', '(DESCRIPTION=(CID=DWH_APP)(ADDRESS_LIST= (ADDRESS=(PROTOCOL=TCP)(HOST=locksnlatches)(PORT=1521)))(CONNECT_DATA=(SID=dwh)))';'scott';'tiger','SELECT 1 FROM DUAL') x

3) Without a linked server (TNS-full)

SELECT * FROM OPENROWSET('OraOLEDB.Oracle', 'dwh.world';'scott';'tiger','SELECT 1 FROM DUAL') x

Power view on analysis services tabular

How to create a Power View report using Sharepoint 2010 and Analysis Services 2012 Tabular as the data source?

0) Prerequisites
A server called "bisql01" with the following services installed:

•   sql server 2012 (mssqlserver = default instance)
•   analysis services 2012 (mssqlserver = default instance)
•   analysis services 2012 tabular mode / vertipaq (astabular = named instance)
•   reporting services (mssqlserver = default instance)
•   sharepoint 2010 installed at https://powerview.locksnlatches.com

All services run under user "cloud\bi01_sql". Work is done locally on this server and refered to as "localhost". Data is loaded from the SQL Server database "CPDV" into the Analysis Services database "CPDM".

INSIDE VISUAL STUDIO 2012

1) create project
new project -> analysis services tabular project
  name: cp_windows7

2) set deploy target of model
rightclick project "cp_windows7" -> properties ->
  server: localhost\astabular
  database: cpdm
  impersonation settings: default

note: there is probably also a default instance of analysis services running; this is the regular multidimensional cube instance like the previous versions of sql server, not the named tabular model instance

note: the cpdm tabular database will be created on deploy

note: there will also be a work tabular database for the current project named after the project, first data source and a guid, e.g. CPDV_BI01_SQL_01e0810e-c8ab-46fd-afe6-098348336a9a. Do not delete this database from the analysis server server!

3) add tables to model
model -> import from datasource -> microsoft sql server:
  server name: bisql01 (or "localhost")
  log on to the server: use windows authentication
  database: cpdv
  use service account,
  select tables/views,
  finish

edit the bism model: model -> model view -> diagram view. If there are no relationships in the source, then they can be added here by dragging and dropping fields from a child to a parent table or vice versa.

change connection type: model -> existing connections -> edit
  use sql server authentication
  name: sa
  password: ***

note: use model -> existing connections, to add tables later on from the same connection

4) set date dimension
view the data: model -> model view -> data view

click the date column, table -> date -> mark as date table

note: mark as date table does not work in the diagram view, only in the data view

note: make sure the date column is unique and continuous

note: a fact table can join to the date dimension on this date field, but also on a unique number field. A date field also needs to be selected in the latter case for "mark as date table" to work.

5) set authorization
models -> roles -> new
  name: reader
  permissions: read

models -> roles -> members -> add
  cloud\bi01_sql
  cloud\bi01_admin (=sharepoint account from which powerview is started)

note: if members are added to this role from management studio to the tabular model instance database then they will be overwritten at the next deploy of the project! So, specify the members in the project.

6) build -> deploy

INSIDE SHAREPOINT 2010

7) create semantic model connection
documents -> new document -> bi semantic model connection ->
  filename: model.bism (this is the name of the document in sharepoint; it can be anything)
  servername: bisql01\astabular
  database: cpdm

note: to fix error "database cannot be found", double check that the reader role has read permissions and that the windows accounts used by Sharepoint are added to that group

8) create report data source to semantic model connection
documents -> new document -> report data source ->
  data source type: microsoft bi semantic model for power vier
  connection string: datasource='https://powerview. locksnlatches.com/Shared%20Documents/model.bism'
  credentials: stored credentials
    username: cloud\bi01_sql
    password: ***
  availability: enable

9) create powerview report
click dropdown arrow of report data source -> create power view report

note: to fix error "User 'DOMAIN\user' does not have required permissions": start IE in "run as administrator" mode and connect using "localhost" iso. the server name.

note: the reporting services logfiles are located here C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\WebServices\LogFiles

Quipu connections

Several ways to create a database connection in the open source data warehouse generator Quipu.

SQL Express

• SQL Server configuration manager -> 
• Protocols for SQLExpress -> TCP/IP: enabled
• TCP/IP -> IP Adresses -> IP All -> TCP Dynamic Ports: empty
• TCP/IP -> IP Adresses -> IP All -> TCP Port: 666
• SQL Server management studio -> SQL Express server properties -> Security -> Server Authentication: SQL Server and Windows Authentication mode
• Create a SQL Server authentication login on the database server and assign it to the database that will be used by the connection, e.g. login: quipu, password: quipu
• If the database runs on the same machine as Quipu, use "localhost" as the host name. Otherwise use the remote host name

Note:

• Do not use the instance name, e.g. using "AdventureWorks;instance=sqlexpress" as database works, but will, depending on the SQL Server version, result in an "unable to get information" error.

The nearby-future of storage

The ioDrive Duo by FusionIO out-performs even multiple SSD's placed in parallel. The 320GB version of the ioDrive has a read/write speed of ~1.5GB/s:

http://www.fusionio.com/products/iodriveduo

Difference with traditional SSD:

• The flash memory is integrated in the PCI card.
• The flash memory has been optimized using a proprietary technology.

Downside:

• Cannot be used as a boot drive.
• Requires a 64-bit operating system.
• The price is about 15,000 euro for 320GB version (ssdeurope.com). Ofcourse, this has to be compared against the cost of buying and operating a SAN.
  

SQL Developer thick/OCI driver

The following error messages may appear when using SQL Developer to select data from a SQL Server table via a (heterogeneous or DG4ODBC) database link on an Oracle 11g DB:

Bigger type length than Maximum
OALL8 is in an inconsistent state

Solution: switch to the thick driver to work around this.

In SQL Developer: tools -> preferences -> database -> advanced -> use oci/thick driver = checked.

If this does not work or when getting the error messages:

no ocijdbc11 in java.library.path
vendor code 0

then:

- Download Oracle Instant Client 11.1.0.7.0 basic lite (easily found with Google, register with Oracle to download)

Note: version 11.2 does not work with SQL Developer!

Note: take the 64-bits version on a 64-bits OS, like Windows7 x64.

- Extract files to c:\ora\instantclient_x64_11.1.0.7.0_basiclite
- Add to environment variable PATH: c:\ora\instantclient_x64_11.1.0.7.0_basiclite
- Create a batch file called sd.bat in the rootfolder of the SQL Developer installation, with the following body:

set ORACLE_HOME="c:\ora\instantclient_x64_11.1.0.7.0_basiclite"
start sqldeveloper.exe

- Double-click sd.bat to start SQL Developer
- Add a connection with Connection type = basic

To make a connection with Connection type = TNS:

- Create folder c:\ora\instantclient_x64_11.1.0.7.0_basiclite\network\admin
- Put these files in the admin folder:

sqlnet.ora
tnsnames.ora

- In SQL Developer: tools -> preferences -> database -> advanced -> tnsnames folder: c:\ora\instantclient_x64_11.1.0.7.0_basiclite\network\admin
- Add a connection with Connection type = TNS

Note the following:

• The solution above even (or especially) works when another Oracle client is already installed. For example, if the OracleXE DB (which, at the time of writing, is 10g) is installed locally, then the Oracle Client 10 is also installed. But this client might not work when trying to connect to a remote Oracle 11g DB
• There is no need to set the environment variables ORACLE_HOME or TNS_ADMIN variable on the system. It does not matter what their values are
• The order of folders in the variable PATH does not matter
• Most likely Help -> About -> Properties -> oracle.home does not point to the correct Oracle home folder. So not to c:\ora\instantclient_x64_11.1.0.7.0_basiclite. It does not matter what the value is
• Unfortunately, the ORACLE_HOME trick does not work with the PATH variable; it is no use adding that to the batch file
• The error "Bigger type length than Maximum" is related to a bug in the implementation of the thin part of the JDBC driver. Not to heterogeneous services, Oracle DB, SQL Server, SQL Developer or the thick part of the JDBC driver. (Note: the same driver, e.g. ojdbc6.jar, contains the thin and the thick classes)

Databases

Row-oriented:
Oracle DB
SQL Server
SQL Server CE
MySql: InnoDB
Sybase Advantage Database Server (ADS)
MS Access

Column-oriented:
Extreme-DB
InfiniDB Enterprise Edition
Infobright
MonetDB
ParAccel Analytic Database
Sybase IQ
Sybase Adaptive Server Enterprise (RDBMS & column-oriented)
Vectorwise
Vertica

Appliance:
Aster Data (MPP, MapReduce)
Greenplum (MPP, MapReduce)
Netezza (MPP)
Teradata (MPP)

In-process:
SQLite

Run SSMS using different Windows account

You can log on to a SQL Server database using SQL Server Management Studio (SSMS) with a Windows Authentication login / NT account / Windows login.

However, the Active Directory domain account to do this with cannot be entered. It is greyed out and always the account of the user that is currently logged on.

There are several ways to bypass this and make a connection:

1) Run SSMS under different privileges

• Create a shortcut to SSMS with the following command:
  runas /netonly /user:domain\usr "g:\Program Files (x86)\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\SqlWb.exe"
• Double-click the shortcut and enter password for the runas account

Note: SSMS can be located somewhere else, depending on the choice made for the location when installing or the operating system. When using Windows 7 x64, then the tool is by default located as specified above.

Note: The greyed-out login screen in SSMS still displays the logged on user, not the runas account. This might be confusing.

1a) Run SSMS under different privileges, without needing to enter a password

• Download "runasspc" (can be easily found with a search engine)
• Extract "runasspc.exe" from the zip file and copy to the windows folder
• Create a shortcut to SSMS with the following command:
  runassprc "g:\Program Files (x86)\Microsoft SQL Server\90\Tools\Binn\VSShell\Common7\IDE\SqlWb.exe" /domain:domain /user:usr /password:pwd

2) Create a network connection to the remote server

• Open a command windows (Win+R, cmd.exe)
• Delete all mappings: net use * /delete /y
• Create mapping to SQL Server server: net use \\servername /user:domain\usr
• Start SQL Server Configuration Manager via Programs (or locate SQLServerManager.msc directly)
• Open SQL Native Client Configuration
• Enable Named Pipes protocol and move to top in the order
• Add a new alias. Use:
  a. "Named Pipes" as network library
  b. Server name: fill in remote server name. The pipe field will display: \\[servername]\pipe\sql\query"
• Start SSMS
  a. Server name: fill in remote server name
  b. Authentication: Windows Authentication
  c. Connection Properties -> Network Protocol, set to Named Pipes

Note: SSMS automatically uses the account name from the connection to the remote server.

3) Set credentials for a remote server using Credential Manager

• Control Panel -> Credential Manager
• Add the domain\account and password for the remote server

Note: use the fully qualified server name and port, e.g. remoteserver.domain.com:1433

4) Set terminal services Active Directory rights
Get the "allow logon to terminal services" right for the remote server for the domain account. Then make a terminal services connection to the server with that account and start SSMS within that session.

Links:
http://www.olegsych.com/2009/05/crossing-domain-boundaries-windows-authentication/

Identifying keys

A key that is made up of one field is a simple key. All other keys are combined keys. There are two types of combined keys, compound and composite:

Simple key: key that consists of one field

Combined key: key that consists of more than one field

Compound key: combined key in which each field is itself a simple key (of another entity)

Composite key: combined key, that is not a compound key

A new term I have made up myself:

Concatenated key: a simple key, that is the concatenation of the fields in a combined key

External tables

Data can be imported into and exported from Oracle 10g in different ways. E.g. by using the SQL*Loader tool for text files (sqlldr) or the Data Pump (impdp/expdp) tools for the propriety Oracle RAW format. For previous Oracle versions, the tools imp and are also available. Those import/export to external locations on the client machine, not the server machine.

Another way to get data in and out of Oracle is to define an external table. To do this you have to create a table in Oracle in the regular way using a CREATE TABLE statement, but with adding an additional ORGANIZATION EXTERNAL clause.

Within this clause the properties of the external (server side!) location can be specified.

EXAMPLE

CREATE TABLE TEST
(
  "CODE" VARCHAR2(17 BYTE)
)
ORGANIZATION EXTERNAL
(
  TYPE ORACLE_LOADER
  DEFAULT DIRECTORY DATA_PUMP_DIR
  ACCESS PARAMETERS (
    RECORDS DELIMITED BY NEWLINE
    CHARACTERSET WE8MSWIN1252
    FIELDS TERMINATED BY 0X'09'
    MISSING FIELD VALUES ARE NULL
    REJECT ROWS WITH ALL NULL FIELDS
  )
  LOCATION ( DATA_PUMP_DIR: 'EXTFILE.TXT' )
)

FIELD LIST

The list of fields following the create table keywords cannot be inferred dynamically, but has to be specified explicitly. So it is not possible to "hook up" a text file and retrieving the list of fields automatically.

Trick: import the data first into a table using SQL Developer, TOAD, etc, and create the DDL script from it. Finally, copy and paste the field list from this script.

It is also possible to specify the "external" list of fields inside the organization external clause. This is, for example, similar to how it is done using Oracle Warehouse Builder (OWB).

TYPE

One of these properties is "TYPE", which specifies the access driver that Oracle uses. There are two values for this property:

• ORACLE_LOADER: only read access, from all kinds of text files (but not, for example, Excel .xls)
• ORACLE_DATAPUMP: read and write access, a propriety RAW format

The access driver is a wrapper Oracle uses to call the previously mentioned SQL*Loader or Data Pump functionality.

An error occurs when an insert statement is performed on a table is created with access driver ORACLE_LOADER:

SQL Error: ORA-30657: operation not supported on external organized table30657.0000 -  "operation not supported on external organized table"
*Cause:    User attempted on operation on an external table which is
           not supported.
*Action:   Don't do that!

The ORACLE_DATAPUMP type has to be used to export data from Oracle using an external table. Unfortunately, this data cannot be humanly read or imported into any other tool then an Oracle database.

DEFAULT DIRECTORY / LOCATION

A directory object has to be available in Oracle. If it is not already present, it has to be created. For example:

CREATE OR REPLACE DIRECTORY DATA_PUMP_DIR AS 'C:\'

Now the tag DATA_PUMP_DIR can be used in the organization external clause. The DEFAULT DICTIONARY property is mandatory and has to be used when setting the LOCATION.

A path specification is not allowed between the single-quotes.

This is not valid:

• LOCATION ( 'C:\EXTFILE.TXT' ) 

ACCESS PARAMETERS

The options are rather straightforward, except perhaps "FIELDS TERMINATED BY". To set the field delimiter a character can be specified between single-quotes. But it is also possible to specify special characters like tab or linefeed.

To specify a tab:

FIELDS TERMINATED BY 0X'09'

The 0X instructs Oracle that the value between the single-quotes should be interpreted as an ASCII value. In the example, ASCII character 9 is a tab.

OPTIONALLY ENCLOSED BY '"'

It is also possible to specify that some values have double-quotes around them. This however does not work in the following case:

Column 1 value - tab - tab - Column 3 value

Here a record has a value for columns 1 and 3, but not for column 2. It should be read as NULL, but is actually skipped. This results in the value of column 3 being wrongly inserted in column 2.

CHARACTER SET

Explicitly setting the character set of the external location prevents problems with interpreting the data and can improve performance.

From Odi's blog:

If you do not specify the CHARACTERSET parameter, the database character set is used to interpret the input file. As the input file is in WE8MSWIN1252, the ORACLE_LOADER driver gets confused trying to interpret single-byte WE8MSWIN1252 codes as multibyte AL32UTF8 codes.

The character set of the input file depends on the way it was created. Even on US Windows, you can create text files in different encodings. Notepad allows you to save the file in ANSI code page (=WE8MSWIN1252 on US Windows), Unicode (=AL16UTF16LE), Unicode big endian (=AL16UTF16), and UTF-8 (=AL32UTF8). The Command Prompt edit.exe editor saves the files in the OEM code page (=US8PC437 on US Windows).

-- Sergiusz

LOGGING

All errors are logged to a bad file - containing the violating records - and a log file - containing messages about the error that occured.

To disable logging, (some of) the following keywords can be added to the clause:

• NODISCARDFILE
• NOBADFILE
• NOLOGFILE

Oracle specification of the ORACLE_LOADER access driver:

http://www.stanford.edu/dept/itss/docs/oracle/10g/server.101/b10825/et_params.htm

SQLite enforces foreign key constraints

As of version 3.6.19, SQLite finally supports foreign key constraints that can be enforced.

http://sqlite.org/releaselog/3_6_19.html

A pragma statement has to be execute on the database to enable the enforcement:

PRAGMA foreign_keys = true;

Measuring disk I/O speed

Disk thruput tester is a handy tool to test the sequential read and write performance of storage media, like harddisks and SSD. It supports configurating custom block sizes:

http://www.objectso.nl/

Using this tool, testing the read/write performance of 8KB, 16KB, 32KB, etc, sized blocks is a breeze. The size and location of the test file is also configurable.

Note: on the website the tool is called "disk throughput tester".

Cursor variants in Oracle 10g

Examples of different types of cursors. Each has its own pros and cons. Example 1 is the most straightforward and easiest to understand. The variants in example 3 perform better. I've put them here for reference.

-- 1) implicit cursor

DECLARE l_old_tbs VARCHAR(100) := 'USERS';
BEGIN
FOR item IN (
SELECT OWNER, TABLE_NAME FROM ALL_TABLES WHERE TABLESPACE_NAME = l_old_tbs )
LOOP
dbms_output.put_line(item.TABLE_NAME);
END LOOP;
END;

-- 2) explicit cursor

DECLARE l_old_tbs VARCHAR(100);
CURSOR c1 IS (SELECT OWNER, TABLE_NAME FROM ALL_TABLES WHERE TABLESPACE_NAME = l_old_tbs);
BEGIN
l_old_tbs := 'USERS'; -- set variable here, otherwise it is NULL!
FOR item IN c1 LOOP
dbms_output.put_line(item.TABLE_NAME);
END LOOP;
END;

-- 3a) local collection variable with multiple fields using BULK COLLECT and FORALL

CREATE TABLE TEST1 (OWNER VARCHAR(30), TABLE_NAME VARCHAR(30));
DECLARE l_old_tbs VARCHAR(100);
TYPE r_tmp IS RECORD (OWNER ALL_TABLES.OWNER%TYPE, TABLE_NAME ALL_TABLES.TABLE_NAME%TYPE);
TYPE t_tmp IS TABLE OF r_tmp;
tmp t_tmp;
BEGIN l_old_tbs:= 'USERS';
-- fill local collection "tmp" with one bulk operation
SELECT OWNER, TABLE_NAME BULK COLLECT INTO tmp FROM ALL_TABLES WHERE TABLESPACE_NAME = l_old_tbs;
-- update/insert/delete with one bulk operation; select not permitted.
-- no "loop" keyword, because following statement is performed as one batch. Check dbms_output, this is only one line.
-- always use "commit" statement
FORALL i IN tmp.FIRST..tmp.LAST
INSERT INTO TEST1 VALUES tmp(i);
dbms_output.put_line('first: 'tmp.FIRST', last: 'tmp.LAST);
COMMIT;
END;

-- 3b) local collection variable with single field using BULK COLLECT and FORALL

DECLARE l_old_tbs VARCHAR(100);
TYPE t_tmp IS TABLE OF ALL_TABLES.TABLE_NAME%TYPE;
tmp t_tmp;
BEGIN
l_old_tbs:= 'USERS';
-- fill "tmp" with one bulk operation
SELECT TABLE_NAME BULK COLLECT INTO tmp FROM ALL_TABLES WHERE TABLESPACE_NAME = l_old_tbs;
-- update/insert/delete with one bulk operation; select not permitted.
-- no "loop" keyword, because following statement is performed as one batch.
-- always use "commit" statement
FORALL i IN tmp.FIRST..tmp.LAST
INSERT INTO TEST1 (TABLE_NAME) SELECT tmp(i) FROM DUAL;
COMMIT;
END;

-- 3c) local collection variable with multiple fields using BULK COLLECT and FOR

SET SERVEROUTPUT ON;
SET SERVEROUTPUT ON SIZE UNLIMITED;
DECLARE
  l_old_tbs VARCHAR2(100);
  TYPE r_tmp IS RECORD (OWNER ALL_TABLES.OWNER%TYPE, TABLE_NAME ALL_TABLES.TABLE_NAME%TYPE);
  TYPE t_tmp IS TABLE OF r_tmp;
  tmp t_tmp;
BEGIN
  l_old_tbs:= 'USERS';
  -- fill local collection "tmp" with one bulk operation
  SELECT OWNER, TABLE_NAME BULK COLLECT INTO tmp FROM ALL_TABLES WHERE TABLESPACE_NAME = l_old_tbs;
  -- update/insert/delete with one bulk operation; select not permitted.
  -- with "loop" statement. Check dbms_output, there are as much lines as tmp.LAST.
  -- always use "commit" statement
  FOR i IN 1..tmp.COUNT
  LOOP
    --INSERT INTO TEST1 VALUES tmp(i);
    dbms_output.put_line('tablename: '||tmp(i).TABLE_NAME||', last: '||tmp.LAST);
    COMMIT;
  END LOOP;
END;

-- 3d) local collection variable with single field using BULK COLLECT and FOR

SET SERVEROUTPUT ON;
SET SERVEROUTPUT ON SIZE UNLIMITED;
DECLARE
  i INT := 0;
  l_tabname VARCHAR2(100);
  TYPE r_tmp IS RECORD (TABLE_NAME USER_TABLES.TABLE_NAME%TYPE);
  TYPE t_tmp IS TABLE OF r_tmp;
  tmp t_tmp;
BEGIN
  -- fill local collection "tmp" with one bulk operation
  SELECT TABLE_NAME BULK COLLECT INTO tmp FROM USER_TABLES WHERE UPPER(TABLE_NAME) LIKE 'TTMP_%';
  -- update/insert/delete with one bulk operation; select not permitted.
  -- with "loop" statement. Check dbms_output, there are as much lines as tmp.LAST.
  -- always use "commit" statement
  FOR i IN 1..tmp.COUNT
    LOOP
      dbms_output.put_line(i);
      l_tabname := 'DROP TABLE ' || tmp(i).TABLE_NAME; 
      dbms_output.put_line(l_tabname);
      EXECUTE IMMEDIATE l_tabname;
      COMMIT;  
    END LOOP;
END;

-- 4) use a temp table

CREATE GLOBAL TEMPORARY TABLE tmp (OWNER VARCHAR(30), TABLE_NAME VARCHAR(30);

Oracle SQL Developer screen redraw glitch

Perform the following steps to fix the redraw, or screen refresh, bug that can occur in Oracle SQL Developer when used on Windows Vista and Windows 7. They occur for example when scrolling horizontally or vertically through the records of a table when the records don't fit the window.

Start with just the sqldeveloper.conf adjustment. If that doesn't fix it, then perform all steps.

Steps to solve the issue:

• Add or replace the attribute of the following config file.
  ..\sqldeveloper\bin\sqldeveloper.conf
  AddVMOption -Dsun.java2d.noddraw=true
• Install the most recent jdk.
  Last seen here: http://java.sun.com/javase/downloads/index.jsp
  Select the Java SE Development Kit (JDK). At this time, JDK 6 Update 16 is the most recent version. Install it in C:\Program Files\Java\jdk1.6.0_16
• Add or replace the attributes of the following config files.
  ..\ide\bin\jdk.conf
  SetJavaHome C:\Program Files\Java\jdk1.6.0_16
  ..\ide\bin\ide.conf
  AddVMOption -Xmx256M
  AddVMOption -Dsun.awt.keepWorkingSetOnMinimize=true

Truncate staging tables

Before creating a backup of the database, you could clean out all staging tables to safe some space in the backup and to reduce the time it takes to make the backup. You could use the following simple SQL-loop statement:

EXEC sp_MSforeachtable "IF LEFT('?',11)='[dbo].[STG_' BEGIN PRINT '?' TRUNCATE TABLE ? END"

The statement expects a strict naming convention: all staging table names should start with STG_ and this prefix should not be used for any other table types. Also make sure you do not have any "permanent" data in your staging environment.

Note:
The PRINT-statement can be left out. It could help you debug the statement. Also the truncate statement does not work if there are foreign keys referencing the table.

Return the size of pre-selected tables

A stored procedure I use a lot to get a resultset containing the size of a selection of tables in the database:

CREATE procedure sp_TableSize as
begin
SET ANSI_WARNINGS OFF

CREATE TABLE #temp
(
TABLENAME varchar(100),
ROWS int,
RESERVED varchar(25),
DATA varchar(25),
INDEX_SIZE varchar(25),
UNUSED varchar(25),
)

DECLARE @table nvarchar(100)

-- Select the tables for which to retrieve sizes
DECLARE cDWHTables cursor read_only for
SELECT name FROM sysobjects
WHERE LEFT(name, 4) in ('STG_','ODS_','DIM_','SYS_','FCT_','RPT_') OR
LEFT(name,3) = 'MD_'

OPEN cDWHTables

-- Store each record retrieved from sp_spaceused in temp table
FETCH NEXT FROM cDWHTables INTO @table
WHILE (@@fetch_status <> -1)
BEGIN
IF (@@fetch_status <> -2)
BEGIN
insert #temp exec sp_spaceused @table
END
FETCH NEXT FROM cDWHTables INTO @table
END

UPDATE #temp SET
reserved = left(reserved, len(reserved) - 3),
data = left(data, len(data) - 3),
index_size = left(index_size, len(index_size) - 3),
unused = left(unused, len(unused) - 3)

INSERT #temp
SELECT 'TOTAL' as tablename, sum(rows) as rows,
sum(convert(int,reserved)) as reserved,
sum(convert(int,data)) as data,
sum(convert(int,index_size)) as index_size,
sum(convert(int,unused)) as unused
FROM #temp

--Return the results as data set
SELECT
TABLENAME = tablename,
ROWS = convert(int,rows),
CONVERT(int,reserved) as RESERVED,
convert(int,data) as DATA,
convert(int,index_size) as INDEX_SIZE,
convert(int,unused) as UNUSED,
AVG_DATA = convert(numeric(18,2),case when rows>0 then convert(float, data) / convert(float, [rows]) else 0 end),
TOTAL = convert(int,data) + convert(int,index_size),
TOTAL_MB = (convert(int,data) + convert(int,index_size))/1024
FROM #temp

-- Clean up
DROP TABLE #temp
CLOSE cDWHTables
DEALLOCATE cDWHTables

end

Replace the bold part to set the tables for which to return the size. After all records have been returned it adds a final total row to the set.

Note:
The insert of the total row can alternatively be written using the COMPUTE-clause on the main select statement.