Home : Tcl/Tk : Namespaces

Namespaces and Packages

Don't shoot yourself in the foot

Putting Tcl library code in a package is a complex process. Tcl 8.x namespaces are confusing. Put the two together, and you've got many new and entertaining ways to shoot yourself in the foot. Worse, the future users of the package are more likely to take the bullet (in the form of confusion and dismay) than the package's author.

The standard Tcl distribution has sufficient documentation about how the namespace and package commands work, but little information about how to use namespaces and packages cleanly and effectively. Nor, in searching the 'net, have I seen anything on how best to use Tcl 8.x namespaces, or on how best to use namespaces and packages together. On the contrary, it's clear from reading comp.lang.tcl and the [incr Tcl] mailing list that there is great confusion in the land.

The following guidelines are some rules I evolved in my own work, as modified by comments and suggestions from other members of the Tcl/Tk community. These guidelines work for me, and keep my packages and programs from stepping on each other's toes. I submit this as a work in progress; I'd greatly appreciate comments, improvements, and arguments as to why it's wrong (if indeed it is).

I assume the reader has access to the Tcl documentation, and perhaps one or more books about Tcl. I'll not explain the individual Tcl commands in any detail, merely how to use them.

The guidelines are broken into two sections:

The Rules
Implementation Issues

The Rules

If you follow these rules, you'll have an easier time getting your package to work properly, and your users will have an easier time using your package.

Rule 1: Follow the Tcl Style Guide.

Except when it makes better sense not to.

The Tcl Style Guide was written by the Tcl/Tk development team at Sun Microsystems, and has become the de facto style guide for the Tcl/Tk community. Like any style guide, its requirements are somewhat arbitrary. It is not intended as a guide to writing code that behaves well, but rather as a guide to writing code that is clear and readable.

These guidelines adhere to the rules set down in the Style Guide, so far as they pertain to issues of naming, code layout, and so forth. Where these guidelines break the rules, they do so explicitly and the reason is stated.

Rule 2: Put library code in a package.

A package is the best way to create a library of Tcl code. Once a package is installed on a system, scripts can use the package without reference to its actual location. Always put Tcl library code in a package.

For example, one might create a package of Tcl code called "myLib":

# myLib.tcl
package provide myLib 1.0

# Package code.

A script can make use of the commands in "myLib" merely by requiring the package:

# In an application or another package
package require myLib

Give the package a name beginning with a lower-case letter; begin the second and following words in the name, if any, with an upper-case letter.

The name should be unique; if your package will be used by many people outside your organization, check the comp.lang.tcl newsgroup, the Tcl websites, and the NIST Identifier Collaboration Service (NICS) at http://pitch.nist.gov/nics.

Chris Nelson suggests using a lowercase name for general utility packages and an initial cap name for packages that are proprietary or too specific to a particular application to be easily reused. This is a generalization of the Tcl Style Guide's convention of lower case for public names and initial cap for private names.

These guidelines will use the package names myLib and someLib as examples.

Practical Programming in Tcl and Tk by Brent Welch, and Effective Tcl/Tk Programming by Harrison and McLennan, have detailed descriptions of how to create a package.

Rule 3: Don't pollute the global namespace.

In versions of Tcl prior to 8.0, and in Tcl 8.x by default, all command and non-local variable names are created in the global namespace. This lead to name collisions between applications and libraries, and was a Bad Thing; hence, namespaces were created.

In Tcl 8.x, the global namespace is the exclusive property of the application. Your package should never place commands or variables in the global namespace (unless asked, of course).

Rule 4: Don't export standard command names.

Unless you really mean it.

For example, don't export "::myLib::set", "::myLib::open", or "::myLib::puts". There are valid reasons to redefine standard Tcl commands, and the language explicitly supports it--but don't do it by accident.

Using a standard prefix for the commands in your package helps here; there's nothing wrong with "::myLib::mySet".

Rule 5: Put the package code in a namespace of the same name.

The namespace should be a child of the global namespace.

# myLib.tcl
package provide myLib 1.0

namespace eval ::myLib:: {
    # code
}

# More code

Some people prefer to give the package name an initial cap and make the associated namespace all lower case. This strikes me as a foolish inconsistency. I claim they should be identical; it's less for the user of the package to remember. It's also consistent with the Tcl Style Guide.

The Tcl Style Guide implies by example that a relative namespace should be used, instead of a fully qualified namespace. There are valid reasons for using a relative namespace instead of one explicitly rooted in the global namespace; it makes it possible to load the package several times into different contexts. It's difficult to get it quite right, unless you are quite sure what you are doing, so I don't recommend it. Explicitly placing your package's namespace in the global namespace means that a naive user of your package is less likely to have problems.

Rule 6: Explicitly require packages used by your package.

If your package uses package someLib, it must not assume that the application will require someLib itself; rather, your package must explicitly require someLib. Otherwise, users of your package will have to play detective when they unknowingly neglect to require someLib in their application code.

Any packages used by your package should be required at file scope, not in a "namespace eval" block.

# myLib.tcl
package provide myLib 1.0

# Right
package require someLib

namespace eval ::myLib:: {
    # Wrong
    package require someLib
}

If someLib's namespace is fully qualified, it doesn't matter which way you do it. If it is relative, though, the second approach will load it into the ::myLib::someLib:: namespace, which is probably a mistake. Require packages at file scope, unless you're sure you know what you're doing.

Note: This last caution is evidently no longer needed; package require now ensures that any code loaded by it is nominally loaded into the global scope. This was pointed out to me by a correspondent, and I'm not sure precisely when the change occurred.

However, I still think that putting the package require commands at file scope is still a good idea.

Rule 7: Define package commands using a fully qualified name.

The Tcl Style Guide requires that package commands must be defined in the package's namespace by defining them at file scope with fully qualified names, e.g.,

# myLib.tcl
package provide myLib 1.0

package require someLib

namespace eval ::myLib:: {
    # Wrong
    proc myBadProc {a b c} { ... }
}

# Right
proc ::myLib::myProc {a b c} { ... }

There is one problem with this approach. If you are creating your procs using something other than the standard "proc" command, e.g., ::tcl::OptProc, or an [incr Tcl] class name, you must use the creation command's fully qualified name, as you are working in the context of the global namespace and cannot import the creation command's name (see below for why this is true). If you define your procs in the context of the "namespace eval", on the other hand, you can import the command's name, and use it without qualification:

#OK
::tcl::OptProc ::myLib::myOptProc {a b c} {...}

namespace eval ::myLib:: {
    namespace import ::tcl::OptProc

    # Also OK, though contrary to Tcl Style Guide
    OptProc myOtherOptProc {a b c} {...}
}

Note that the Tcl Style Guide says that public names should begin with a lower case letter; private names should begin with an uppercase letter.

Rule 8: Explicitly export public commands.

Your package should explicitly export its public commands, and should never export its private commands. This allows users of your package to "namespace import" the public commands using a wildcard if they wish. In addition, if your package is auto-loaded (see Implementation Issues), only the exported commands are placed in the auto-loader's index.

The public command names can be exported individually; alternatively, if you are lazy and follow the "Initial cap = private/ initial lower case = public" convention from the Tcl Style Guide, you can use a wild card.

If the package contains this code,

namespace eval ::myLib:: {
    # The lazy way
    namespace export {[a-z]*}

    # The proper but more tedious way, if you have many commands to export
    namespace export myPublicProc
}

proc ::myLib::myPublicProc {} { ... }

proc ::myLib::MyPrivateProc {} { ... }

the caller can use this code:

namespace import ::myLib::*

This will import myPublicProc but not MyPrivateProc.

There is more about importing names, below.

Rule 9: Define package variables using the `variable` command.

In pre-namespace days, a Tcl library module would typically define one global variable, an array, and use it to store all of the module's private data. This is no longer necessary with namespaces. The "variable" command defines variables within your package's namespace.

If the variable must be initialized when the package is loaded, initialize it in a "namespace eval" block. Otherwise, you can just use "variable" to declare it in the procedures that need it. As a matter of form, it is useful to declare all variables in the initial "namespace eval" block, even if they need no initialization, as it acts as documentation for the package's data. It is helpful to comment each variable declared in this way.

# myLib.tcl
package provide myLib 1.0

package require someLib

namespace eval ::myLib:: {
    # MyVar always contains a meaningless string.
    variable MyVar "Initial value"
}

::myLib::myProc {a b c} {
    variable MyVar
    variable MyOtherVar

    # more code
}

Note that the Tcl Style Guide says that public names should begin with a lower case letter; private names should begin with an uppercase letter.

Rule 10: Be careful changing `global` to `variable`

If you are putting older library code into a namespace, and switching from using "global" to using "variable", don't forget that the two commands have different parameters. For example,

global Foo Bar

declares two global variables, "Foo" and "Bar". On the other hand,

variable Foo Bar

declares a namespace variable, "Foo", and assigns "Bar" to it.

Rule 11: Pass fully-qualified variable names to other packages.

In your package, if you pass a private variable name to some other package, e.g., to a Tk widget, be sure to give use its fully-qualified name.

namespace eval ::myLib:: {
    variable MyVar "The quick brown fox, etc."
       
    # Wrong: .lab1 will use ::MyVar
    label .lab1 -textvariable MyVar

    # Right:
    label .lab2 -textvariable ::myLib::MyVar

    # Also Right
    label .lab3 -textvariable [namespace current]::MyVar
}

The second and third forms are equivalent in this case, since package myLib always fully qualifies its namespace. If you should use a relative namespace, then the third form is better, as it is guaranteed to be correct.

Variables passed to another package, like Tk, are generally evaluated in the global scope; if the variable isn't fully qualified, you'll either get an error because the variable doesn't exist, or your widget will be displaying a variable other than the one you intended.

Rule 12: Pass scripts to another package using `namespace code`

In your package, if you pass a script or a command to some other package, e.g., to a Tk widget, wrap it in "namespace code".

namespace eval ::myLib:: {
    # Wrong: .btn will call ::Doit, not ::myLib::Doit
    button .btn1 -text DoIt -command Doit

    # Right:
    button2 .btn -text DoIt -command [namespace code Doit]
}

proc ::myLib::Doit {} {...}

A package should always evaluate callbacks in the global scope. As a result, when .btn1 is pushed you'll either get an error or call the wrong command.

Rule 13: Never import names into another person's namespace.

The "namespace import" command allows a user of a package to import one or more of the package's command names into the current namespace. If the same name is imported twice, "namespace import" throws an error. This protects the caller from importing two commands with the same name, but from different packages. It also means that you can't import the same command twice from a single package. (This is a bug, in my view, but perhaps it can't be helped.) It's possible to force importation, but then why use namespaces at all?

Anyway, if your package imports a name into a namespace that doesn't belong to it (such as the global namespace), and the owner of that namespace tries to do the same thing, problems will arise. Don't do it.

Rule 14: Import names into your own namespace.

If your package uses another package, and imports any of the names from the other package, it should import them into its own namespace:

# myLib.tcl
package provide myLib 1.0

package require someLib

# Wrong:
namespace import ::someLib::*

namespace eval ::myLib:: {
    # Right:
    namespace import ::someLib::*
}

This way, other packages, or the application itself, can import someLib's names into their own namespaces without error.

Some say you should never import names; I disagree. If a package contains at most one or two commands that are used rarely, it makes sense not to import the names. A megawidget package, for example, probably contains one public command to create the megawidget. After that, you use the widget command, which is in the global namespace by convention. (This is an example of putting a name in the global namespace when asked to do so.) If a package contains many commands, and your code depends heavily on them, then I claim it makes a lot of sense to import the names.

Note that in versions of Tcl 8.0 prior to Tcl 8.0.3, there was a bug in the "namespace import" command. If a package was defined with autoloading (see Implementation Issues, below), you couldn't import any names from it until the package file which defined the names had really been sourced. In short, to import names from the package's namespace, you first had to arrange for the package to be completely loaded, usually by using one of the package's commands with a fully qualified name. This bug has been fixed in Tcl 8.0.3; this is yet another reason to move up to Tcl 8.0.3 if you're using an earlier version of Tcl 8.0. Credit to Mike Gahan for pointing out the problem, and to Brent Welch for telling me that it's been fixed.

Rule 15: An application should explicitly require the packages it uses.

If the main program of your application uses a package directly, it should explicitly require it, even if some other package used by the application requires it as well. It does no harm to require a package more than once.

package require myLib
package require someLib

Here, myLib requires someLib, but that's all right.

Rule 16: An application may import names into the global namespace.

The application owns the global namespace, and therefore may import any names that it chooses:

package require myLib
package require someLib

namespace import myLib::* someLib::*

In this example, someLib's names are imported into the global namespace by the main program, and into myLib's namespace by myLib. There is no conflict.

Lots of people have had problems with [incr Tcl] 3.0 and [incr Widgets] 3.0; partially, this is because [incr Widgets] breaks the above rules. For example, it assumes that the "class" command has been imported into the current namespace.

Implementation Issues

This section discusses a number of issues relating to packages and namespaces for which I wasn't able to formulate hard-and-fast rules. Developers will have to make up their own minds.

Issue 1: Auto-loading vs. Immediate Loading

One a package has been written, it requires a pkgIndex.tcl file. The packaging loading code looks for this file, and uses it to create an index of available packages. When a package is required, Tcl can use the index to load the package.

Conventionally, as described in Practical Programming in Tcl and Tk by Brent Welch, one uses the "pkg_mkIndex" command to create the pkgIndex.tcl file. The file looks like this:

# Tcl package index file, version 1.1
# This file is generated by the "pkg_mkIndex" command
# and sourced either when an application starts up or
# by a "package unknown" script.  It invokes the
# "package ifneeded" command to set up package-related
# information so that packages will be loaded automatically
# in response to "package require" commands.  When this
# script is sourced, the variable $dir must contain the
# full path name of this file's directory.

package ifneeded myLib 1.0 [list tclPkgSetup $dir myLib 1.0 \ 
        {{myLib.tcl source ::myLib::proc1 ::myLib::proc2}}]

The above code assumes that the myLib package contains two public commands, "proc1" and "proc2". When package myLib is required, the "tclPkgSetup" command will add these commands to the auto-loader's index, so that the first use of either one will cause myLib.tcl to be sourced. Thus, requiring the package does not actually load any of the package's code. This is a good thing in a number of settings:

If the package is large, with many separate modules (such as [incr Widgets]), only the modules which are really needed get loaded into memory.
In a GUI application, it means that the user interface can be built and displayed quickly; other code will be loaded later on. The total cost of loading the code is the same, but much of it can be loaded without the user noticing.
It is the standard way of creating the package.

On the other hand, auto-loading has a number of disadvantages as well:

In Tcl 8.0, Tcl 8.0p1, and Tcl8.0p2, command names could not be imported from a required package until the package was completely loaded. If one package uses another and imports names from it, this can be a source of mysterious bugs.

Clif Flynt sent me a work-around for this, however. After requiring the package, use the following code:
```
           global auto_index   ; # If this is in a procedure
           eval $auto_index(packageProcedureName)
       
```
where packageProcedureName is the name of some command in the package, e.g., "::myLib::proc1". This code has the effect of sourcing the file in which the command is defined, after which its name can be imported.
If the package has more than one source file, package initialization code becomes a problem: it's unpredictable which file will be sourced first. There are really only four ways to handle it:
1. Put all public commands and the initialization code in a single file. That file may source additional files containing private code.
  
  This is the method recommended by the Tcl Style Guide. I don't like it, as it separates callback functions and other low-level routines from the public routines they implement. In addition, it's an unreasonable constraint for large packages like [incr Widgets].
2. Make each source file independent. Each contains a "package provide" statement, requires the packages it needs, exports its own public commands, and does its own initialization.
  
  This is better if the files really can be independant. On the other hand, it means that none of the files can import any names from other packages. I don't like that, either.
3. Make each source file independent, but put all initialization code (such as importing names) into a single file. Write the file so that sourcing it multiple times has no effect. Have each independent source file source the initialization file.
  
  This is ugly and annoying. I might prefer to forgo importing command names.
4. Break the package into independent packages, each with its own namespace. This works well, but isn't always feasible. If the [incr Widgets] package were broken up this way, for example, it would consist of a dizzying array of packages and namespaces.

However, it's easy enough to write the pkgIndex.tcl file by hand, so that the package's code is loaded immediately. For example:

package ifneeded myLib 1.0 \ 
    [list source [file join $dir myLib.tcl]]

Simply, when myLib is required, myLib.tcl is sourced. This approach has a number of advantages:

There are no problems with importing names, regardless of the Tcl version.
The package can be broken into files in any way the suits the implementor. myLib.tcl may define the whole package, or it may do initialization and source a number of other files.
It is never necessary to regenerate the pkgIndex.tcl file. It is easy, when adding commands to a package, to fail to reindex the package. This can easily be overlooked for quite a while, until someone tries to call one of the new functions. With this technique, though, the index never changes.

I've used this approach quite successfully for proprietary test libraries. Since they are used only in test scripts, a start-up spike isn't a problem, and since the packages need never be re-indexed, they are more robust.

Issue 2: Package-loading during Development

The "package require" command will only find a package if it's in an expected place: a directory or subdirectory of a directory that is named in the auto_path variable. The parent of Tcl's own library directory is always on the auto_path, as is any directory listed in the TCLLIBPATH environment variable. Packages are usually installed by putting them in a well-known directory on the auto_path, so that scripts don't need to know where they are.

During development of a package, though, you probably don't want to work on the installed copy, or install it before it's ready. As a result, you must updated the auto_path somehow. Here are several approaches:

Add your working directory to your TCLLIBPATH. This works very well, but it's a nuisance if you're changing it frequently.
Explicitly add your working directory to the auto_path. This is fine if you're testing a script, but a nuisance when you're testing commands interactively.
Here's a neat trick (credit Chris Nelson); put the following code in your .tclshrc and .wishrc files:
```
       set auto_path [linsert $auto_path 0 .]
       
```
When you test interactively, the current directory is automatically added to your auto_path, allowing your package to be required.

Bibliography

Effective Tcl/Tk Programming: Harrison and McLennan, 1998, Addison-Wesley, ISBN 0-201-63474-0.
Practical Programming in Tcl and Tk, 2nd. Ed.: Brent B. Welch, 1997, Prentice Hall, ISBN 0-13-616830-2.

Acknowledgements

I'd like to thank Chris Nelson, Chang Li, Mike Gahan, Clif Flynt, and Brent Welch for their comments and suggestions. When I first published this for the community's perusal, with fear and trembling, it was with the expectation that it would change, perhaps drastically, as my understanding was upgraded. And so it has, in form, and partially in content, and it is largely due to those listed above that this is the case.

If you have questions, comments, or suggestions about this page, feel free to contact me at will@wjduquette.com.