Developer42

2011-12-09

SQL Optimisation :: Convert Primary Keys to Clustered Indexes

We recently spotted that one of the systems we use did not make use of clustered indexes on any tables out of the box. As a result performance was not as good as it could have been. The below script allows for an easy win optimisation by finding all tables which do not include a clustered index, and converting the table’s primary key to be a clustered index.

In a future post I’ll put up more details on what clustered indexes are, why you should always (pretty much) use them and other useful info for anyone playing with databases.

--This script is designed for MS SQL Server
use DbNameToOptimise
go

declare @sqls table(object_id bigint, sort int, sql nvarchar(max))

insert @sqls
select t.object_id, ic.key_ordinal, case when ic.key_ordinal=1 then 'CREATE UNIQUE CLUSTERED INDEX [' + i.name + '] ON [' + t.name + ']([' else ',[' end + c.name + case when ic.key_ordinal=icagg.maxko then ']) WITH DROP_EXISTING' else ']' end sql
from sys.tables t
inner join sys.indexes i on t.object_id = i.object_id
inner join sys.index_columns ic on i.object_id=ic.object_id and i.index_id = ic.index_id
inner join sys.columns c on ic.object_id = c.object_id and ic.column_id = c.column_id
inner join (
select object_id, index_id, MAX(key_ordinal) maxko from sys.index_columns group by object_id,index_id
) icagg on i.object_id = icagg.object_id and i.index_id = icagg.index_id
where t.is_ms_shipped=0
and i.is_primary_key=1
and not exists (
--ignore tables which already have a clustered index
select 1
from sys.indexes i2
where t.object_id = i2.object_id
and i2.type = 1
)
order by t.name, i.name, ic.key_ordinal

declare @objid bigint
, @sql nvarchar(max)

while exists (select top 1 1 from @sqls)
begin
set @sql=''
select top 1 @objid=object_id from @sqls
select @sql = @sql + sql from @sqls where object_id=@objid order by sort
delete from @sqls where object_id = @objid
exec (@sql)
end

2011-06-28

A Request / Random Thoughts

As with many of my posts, this is basically an unedited brain dump – apologies. Hopefully this will encourage some interesting comments / discussion though. . .

A Standard for Developer APIs
Facebook, Twitter, Worpress (and I’m sure Google+ will) offer APIs to developers allowing them to pull data out of their applications and manipulate it as they like. Most of these services offer similar functions; authentication, get the last x posts, pull back a grid of contacts, etc. All do this in their own way.
What would be great is some unification – either a library over the top of the existing APIs to pull them all in line, or for some set of social standards to be formed in the same way Netscape, IE, Mozilla and more came up with ECMAScript as a way to allow javascript to become portable. What I’m hoping for is something like this:

//this is entirely made up code - not (yet) some awesome new Google thing
var application = GetApplication('Google+'); //creates a new object with an "application interface" for Google+
if (application.authenticate('Developer42','DemoCodePassword') { //authenticate a user against the web app
var identity = application.me(); //pull back an object which represents me
var allFriends = me.ListContacts(); //by default pull back all contacts
var colleagues = me.ListContacts('colleagues'); //or filter by group
var posts = identity.GetPosts(20); //get my last 20 posts
var friendsPosts = allFriends.GetPosts(100); //Get the last 100 posts by my friends/contacts
}

A Service for Services
This is probably what the guys who came up with UDDI were thinking:

If two companies offer a service to give out exchange rate information, and both use the same standard, when I want to get back exchange rate info why can’t I just post a request to the web saying “give me the exchange rate from USD to GBP” and have it chuck back .67 without all the hassle of searching for a suitable service.

There’s a whole bunch of data which we often need, but have to trawl the web for. Search engines began to make this better, WolframAlpha got a bit closer, but no one’s yet cracked it. What I’d like is a single web site containing a catalogue of services and their schemas. I pick a service, write code to its schema, then use the service url to pull back this data. From my point of view I’m just pulling data from http://www.UsefulServices.com/ExchangeRates, but in the background that could be talking to any (approved) provider. I guess the reason this doesn’t yet exist is the issue around monetisation; but surely there’s a way. . . ?
Below’s my wish list of services:

- Exchange Rates
- Share Prices
- National Holidays
- Daylight Savings Dates
- Post Code / Geo (long & lat) Conversion
- Credit Checks
- Product Prices
- Companies House Info

2011-03-28

Button to spell check a web-page in Google Chrome

Filed under: Google, Technology — Developer42 @ 20:32

A question recently popped up on the Chromium discussion forum asking if there was a way to get Chrome to apply its spell check to a whole website, as opposed to just a text box. Though I don’t know of one, I thought of a little javascript workaround. Copying and pasting the code below into your address bar will copy the text content of the website into a textarea causing Chrome to automatically spell check the contents. Not ideal, but nice for anyone who enjoys scripts.

javascript:var ta=document.createElement('textarea'); var s=document.createAttribute('style'); s.nodeValue='width:100%;height:100em;'; ta.setAttributeNode(s); ta.appendChild(document.createTextNode(document.body.innerText)); document.body.appendChild(ta); ta.focus(); for(var i=1;i<=ta.value.length;i++)ta.setSelectionRange(i,i);

Useful resource: Getting & setting the caret (text cursor) position
Original Question: Q. Button to spell check a web-page

2010-06-29

My First Python Plugin

Filed under: Technology, WordPress — Tags: , , , , , , , , — Developer42 @ 22:51

I’ve read a lot about Python in the past, but have been so inundated with things to learn recently that it had always fallen to the bottom of the pile. However, recently I requested a feature for my favourite text editor, Programmers’ Notepad, and heard back from its developer, Simon, within moments of submitting my request. He’d provided a solution and instructions on how to implement it, which introduced me to a powerful set of features I’d previously been unaware of (namely that you could run apps which output to the command line straight from the tools menu, and have their output returned into the current document). After realising I’d been missing out on some of this potential, I spent a bit of time going over the pnotepad.org site, seeing what else I’d missed. This is where I come back to Python. PNotepad allows you to write custom scripts in Python, put them into pn’s \scripts directory, and run them from the scripts toolbar the next time you start up the editor. I figured any time spent learning python would easily be offset by the hours saved by having scripts take care of laborious tasks for me, so promptly downloaded and installed Python and PyPN (which enables Python scripts in PN), had a look at a script from the script repository, and a browse through the python manual, and then wrote my first script in a few minutes. For anyone wanting to see it, my script is a simple tool for replacing < > and & characters with their HTML codes, and inserting the PRE and CODE tags required to publish code to wordpress. I’ve uploaded it to the script share, here.

First Impressions
My first impressions of Python, after having coded for all of 20 mins or so, are that it’s a really simple language to learn, which anyone (including non developers) should be able to pick up pretty quickly. There are a few bits I’m not so keen on; select statements don’t exist, using instead piles of else-if (elif) statements, and not having curly brackets and semi-colons, but instead relying on indentation leaves me feeling a little out of my comfort zone, but all this is just a familiarity thing, which I’m sure I’ll soon get over. I’m hoping to write a few more scripts over the next few weeks (assuming I can find interesting and required things to write), and to learn a bit more about Python outside of PN, after which I’ll report back on my experiences for anyone else interested in heading in that direction.

Steps Taken to Run Python in PN
If you’d also like to get into Python for PN, here’s a step by step set of instructions on what to do (skipping a few steps to avoid typing “click next” too many times).
Download & install/extract the following, in the order listed below:

Once installed, go to the command line (windows key + r, cmd, enter), navigate to the programmer’s notepad directory (cd %programfiles%\programmer*), then type the following “pn –findexts”. It looks like nothing’s happened, but that’s good; it hasn’t errorred.
Next, load up programmer’s notepad, click view, windows, scripts (or press alt+F10), and you should see a list of available scripts.
Get an existing script from the script share site: http://scriptshare.rocketmonkeys.com/ and save the file to your …\Programmer’s Notepad\scripts\ directory.
Close and reopen PN. You should now see a new script (or several) listed in the scripts toolbar.
From here on, it’s pretty easy to figure out what to do to create your own scripts by looking at existing scripts and python code, and playing.
Good luck.

Grouping Items using XSLT

Filed under: Technology — Tags: , , , , , , — Developer42 @ 12:16

Another XSLT problem. How do you group items together using XSLT, when the sort function isn’t available (e.g. the Muenchian Method won’t work).

The XSLT

<xsl:stylesheet version="2.0"  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

  <xsl:output method="xml" indent="yes" encoding="utf-8" /> <!-- keeping utf 8 rather than 16 as this will be big -->
  <xsl:strip-space elements="*"/>

  <!-- ROOT -->
  <xsl:template match="/node()">
    <xsl:copy>
      <xsl:copy-of select="@*"/>
      <xsl:for-each select="./node()[not(journalName = preceding-sibling::Row/journalName)]/node()[local-name()=&quot;journalName&quot;]"><!-- for each distinct JournalName -->
        <xsl:variable name="journalName"><xsl:value-of select="."/></xsl:variable>
        <Journal>
          <xsl:copy-of select="."/>
          <xsl:apply-templates select="//node()[local-name()=&quot;Row&quot; and journalName=$journalName]" />
        </Journal>
      </xsl:for-each>
    </xsl:copy>
  </xsl:template>

  <!-- By default, copy everything as is -->
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
  </xsl:template>

  <!-- but as we already have the journal name, strip it -->
  <xsl:template match="node()[local-name()=&quot;journalName&quot;]" />

  <!-- and demo that we could manipulate other elements if needed -->
  <xsl:template match="node()[local-name()=&quot;otherData&quot;]">
    <xsl:copy>
      <xsl:copy-of select="@*" />
      <xsl:attribute name="newAttribute">Demo</xsl:attribute>
      <xsl:value-of select="."/>
    </xsl:copy>
  </xsl:template>

</xsl:stylesheet>

Sample Data

<Root>
  <Row>
    <journalName>a</journalName>
	<otherData>hello1</otherData>
	<other>hellox1</other>
  </Row>
  <Row>
    <journalName>a</journalName>
	<otherData>hello2</otherData>
	<other>hellox2</other>
  </Row>
  <Row>
    <journalName>b</journalName>
	<otherData>hello3</otherData>
	<other>hellox3</other>
  </Row>
  <Row>
    <journalName>b</journalName>
	<otherData>hello4</otherData>
	<other>hellox4</other>
  </Row>
  <Row>
    <journalName>a</journalName>
	<otherData>hello5</otherData>
	<other>hellox5</other>
  </Row>
</Root>

2010-06-24

Getting Todays Date from an XSLT Transform

Filed under: Microsoft, Technology — Developer42 @ 17:38

Today I hit an issue whereby I needed to put the current date into an XML document, but found that the current-datetime() function wasn’t supported by my xsl processor. As a result, I’ve written the below code to get around the issue (heavily based on resources found elsewhere on the web). Please see the comments inline for information on what each bit’s doing.
One extra point is that this will only work if scripts are available to your XSLT processor. If you’re using C#, please lookup XsltSettings.EnableScripts for more info.

<!-- int-ap-007_01 2010-06-24 John Bevan -->
<xsl:stylesheet version="2.0"  xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:jsGoodies="urn:internal:jsGoodies">
  <!-- nb: the msxsl and jsGoodies namespaces above are required for the date function.  The ms one allows me to use scripts, the jsgoodies references my script, below -->
  <!-- dont indent; improves processing speed - but leave as yes for debugging - also strip all whitespace if not debugging, as this will help speed and file size -->
  <xsl:output method="xml" indent="yes" encoding="utf-16" />
  <!-- <xsl:strip-space elements="*" /> -->
< put the functions result into a variable to save reprocessing each time its referenced >
  <xsl:variable name="now" select="jsGoodies:GetCurrentDateTime()" />
  <xsl:template match="/node()">
    <xsl:copy>
      <xsl:copy-of select="@*"/>
          <xsl:element name="Now">
        <!-- use xsl:element instead of just &amp;lt;Now&amp;gt; to avoid getting namespace fluff -->
          <xsl:value-of select="$now"/>
          </xsl:element>
      <xsl:apply-templates select="*"/>
    </xsl:copy>
  </xsl:template>

  <!-- By default, copy everything as is -->
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@* | node()"/>
    </xsl:copy>
  </xsl:template>

    <!--
    current-date() is not available in the current MS version of XSLT.  Therefore using a JScript hack

http://social.msdn.microsoft.com/Forums/en-US/sqlintegrationservices/thread/d8c0b688-e176-4316-b70a-637e2bfcf6dc/

    -->
    <msxsl:script language="JScript" implements-prefix="jsGoodies">
      <![CDATA[
        //cdata tells xml parser to ignore this bit, allowing me to do things like this "i > u" & that <--.  Very useful when writing code :)
        function GetCurrentDateTime()
        {
         var result = new Date();
         return ToXmlDate(result);
        }
        function ToXmlDate(d)
        {
          //I'm using the UTC date to avoid timezone issues
          return d.getUTCFullYear()
          + '-' + ToMonth(d.getUTCMonth())
          + '-' + LPad(d.getUTCDate(),'0',2)
          + 'T' + LPad(d.getUTCHours(),'0',2)
          + ':' + LPad(d.getUTCMinutes(),'0',2)
          + ':' + LPad(d.getUTCSeconds(),'0',2)
          + '.' + LPad(d.getUTCMilliseconds(),'0',3)
          + 'Z';
        }
        function ToMonth(i)
        {
          i++; //the month is given as a zero based index, rather than the month value
          return LPad(i,'0',2);
        }
        function LPad(s,c,l)
        {
          return (""+s).PadLeft(l,c); //ensures that s is a string, then uses the strings padleft function
        }
    ]]>
    </msxsl:script>

</xsl:stylesheet>

2010-04-29

Intelligently Splitting an Address

Filed under: SQL Server, Technology, Uncategorized — Tags: , , , , , , , , — Developer42 @ 08:05

An issue recently arose at work where I was asked to take a street address from one field, and split it into two for use with another system. The simple solutions available were:

  • simply splitting on the first comma to appear in the string, and putting each side of this split into the respective fields.
  • breaking the string down the middle.
  • putting everything I could into the first field, then any remainder into the second.

However, none of these seemed to feel right; the first solution would work well, but didn’t take the destination fields’ sizes into account, and the second and third solutions could break half-way through a word / in the case of the third, may not use the second field.
As a result, I came up with the following function; it’s probably not the best solution (since this was written in half an hour on getting home from a night out, and with just a quick check that all worked OK this morning), but hopefully it’ll be useful to some people.

The function takes 3 arguments; the string to be split, the length of the first field to be populated, and the length of the second. It returns a table with an ID field to give the order of the elements (not really required), and a String field to give the two return values in separate rows. This function will always return exactly 2 rows.

The function is written in T-SQL for SQL Server, though I believe it should also work with Sybase.


SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

CREATE FUNCTION dbo.SplitSpecial

(

      @stringToSplit nvarchar(max)

      ,@len1 int

      ,@len2 int

)  

RETURNS @returnVal table (id int identity(1,1),string nvarchar(max)) 

AS  

BEGIN 

      declare @i int = 0                        --used as a counter when searching for chars

      , @j int = 0                              --another char counter

      , @ldiff int = 0                    --says how many characters we have spare to play with

      , @mark int = 0                     --a marker

      , @tempmark int   = 0                     --a marker

      , @back nvarchar(max)               --stringToSplit in reverse

      , @temp nvarchar(max)               --for storing works in progress

      , @forward bit = 0                        --used to indicate the direction of the string from the marker

      --remove any starting, ending or duplicate spaces should they exist, to try to cram in the characters

      select @stringToSplit = ltrim(rtrim(@stringToSplit))

      while (charindex('  ',@stringToSplit)>0)

            select @stringToSplit = replace(@stringToSplit,N'  ',N' ')

      --find out what we have to play with / reduce max size of strings to size of input

      select @tempmark = len(@stringToSplit)

      select @len1  = 0.5*((@len1 + @tempmark) - abs(@len1 - @tempmark))

            ,  @len2  = 0.5*((@len2 + @tempmark) - abs(@len2 - @tempmark))

      select @ldiff = (@len1 + @len2) - @tempmark

      if @ldiff > 0

      begin

            --get a good starting point / break point: no longer required

            --select @mark = round(len(@stringToSplit)/2,0) --assume we can get away with halving the string

            --select @mark = 0.5*((@mark + @len1) - abs(@mark - @len1)) --account for first field being too short (least function math hack)

            --select @mark = len(@stringToSplit) - 0.5*(((len(@stringToSplit)-@mark) + @len2) - abs((len(@stringToSplit)-@mark) - @len2)) --account for 2nd field being too short (least function math hack)

            --the two fields are big enough for us to try some funky stuff with

            --is there a delimeter we can use?

            select @temp = substring(@stringToSplit,len(@stringToSplit)-@len2,@ldiff+1)--get crossover chars; the +1 accounts for dropping the delim

            while patindex('%[a-z0-9]%', @temp) > 0

                  set @temp = stuff(@temp, patindex('%[a-z0-9]%', @temp), 1, '')

            if (len(@temp)+charindex(N' ',@temp)) > 0 --a suitable delim exists in the crossover area

            begin

                  if charindex(N',',@temp)>0 --if commas are available, these are the preferred delimeter

                        select @temp = N','

                  else

                        if charindex(N' ',@temp)>0  --spaces are the second favourite

                              select @temp =N' '

                  select @back = reverse(@stringToSplit)

                  , @i=1

                  , @j = ceiling(len(@stringToSplit)/2)-1

                  , @forward = 1

                  , @mark = len(@stringToSplit)

                  while @i<=(len(@temp)+charindex(N' ',@temp))--try out all available delimeters (if comma or space are present, others are ignored)

                  begin

                        while @j>0

                        begin

                              select @tempmark = charindex(substring(@temp,@i,1),@stringToSplit,@j)

                              if @tempmark > 0 and @tempmark < @mark

                                    select @mark = @tempmark, @forward = 1

                              select @tempmark = charindex(substring(@temp,@i,1),@back,@j)

                              if @tempmark > 0 and @tempmark < @mark

                                    select @mark = @tempmark, @forward = 0

                              if(@mark<=@len1 and (len(@stringToSplit)-@mark)<=@len2)

                                    break

                              select @j = @j - 1

                        end

                        select @i = @i + 1

                  end

                  --select @mark = case @forward when 0 then len(@stringToSplit)-(@mark+1) else @mark end

                  select @mark = case @forward when 0 then (1+len(@stringToSplit))-@mark else @mark end

            end

      end

      --catch any issues & account for when we can't find a good break point / there isn't space

      if (@mark <= 0 or @mark > @len1)

            select @mark = @len1

      insert into @returnVal (string)     values (ltrim(rtrim(substring(@stringToSplit,1,@mark))))

      insert into @returnVal (string)     values (ltrim(rtrim(substring(@stringToSplit,1+@mark,@len2))))

      return

END

GO

--examples of the function in action

select * from dbo.SplitSpecial('33 Rue De Maisons, Paris',100,100)

select * from dbo.SplitSpecial('33 Rue De Maisons, Paris, France 123456',100,100)

select * from dbo.SplitSpecial('33,Rue,De,Maisons,Paris,France,123456',100,100)

select * from dbo.SplitSpecial('33 Rue De Maisons Paris France 123456',100,100)

select * from dbo.SplitSpecial('33 Rue De Maisons, Paris, France 123456',8,50)

select * from dbo.SplitSpecial('this string is too big for the fields',10,10)

At some point, I hope to update this to allow it to take any number of fields of different lengths, to make it more general (e.g. should you ever have a full address to be split into 4 or 5 lines), but for now, it works (so far, at least)

2010-04-10

2010 Scripting Games

Filed under: Microsoft, Technology — Tags: , , , , — Developer42 @ 15:40

I’ve just discovered a coding competition for people who write small but useful utility scripts. Since I my projects tend to get affected by my short attention span, but I enjoy writing useful & helpful stuff, this seems the ideal arena for my work. If you’re also interested in the games, please see the site, linked below:

2010 Scripting Games

Grab this badge here!

2009-11-23

Google Wave Robot :: Standard Commands

Filed under: Google, Technology, Wave — Tags: , , , , , , , , , , — Developer42 @ 19:57

Currently each programmer writing a bot for Google Wave is able to write whatever they want, sometimes providing support, but using their own, custom syntax. Having a set of commands which all (or many) bots implement would make the user experience far better, as users would then be able to easily find out how to use the bot, or where to find additional help.
e.g.
botName@appspot.com /? //see /help
botName@appspot.com /help //brings up a list of commands available
botName@appspot.com /about //displays a summary of what this robot does
botName@appspot.com /devsite //provides a link to the developer’s site
botName@appspot.com /mode private //the bot’s responses are only shown to the person who invited it
botName@appspot.com /mode public //everyone in the wave sees the bot’s responses

I’ve also made a summary of this post available on the Google Wave Bots site. Please can you post any responses to that site, as this will allow all communications to be kept in the same place.

2009-11-08

Psychic Site

Filed under: Ideas, Psychology, Technology — Tags: , , , , , , , , — Developer42 @ 16:35

One of the many projects I have queued up to code one day is a psychic site. This isn’t going to be any weird paranormal thing, or a way of ripping people off, but a way of showing what’s possible. The idea is to create a site, where a user fills in a few details, then the site does a cold reading. However, rather than just using the information provided, the site’s able to use information the person didn’t know they were giving to make it appear to be doing the impossible. Useful tools to aid with this are linked below:

Gender Genie. This shows how the language a person uses gives some indication as to their gender.
Polite Software’s Web Spy. This is a security flaw I discovered a while back, which it seems many geeks have worked out for themselves, but is still not that well known. Since you can tell a visited link from an active link by taking advantage of the different CSS style, and reading this using javascript, you can tell which popular sites a person’s been to. You can then use this information to build up a profile of the person, based on the sites and their demographics (e.g. someone who visits facebook is likely to be under 30, whilst someone going to the bbc news site is likely to be over 20).
HTML 5 GEO. HTML 5 introduces an amazing feature; your standard web browser is able to tell your location without requiring GPS. This is done by using GPS where available, your wireless network card to detect signals from various mobile phone masts, then using the strength of each of these signals along with signal strength maps to triangulate your location, your IP address to find the location of your ISP and various other methods. More on how this works can be found here W3 Geo Spec. Getting hold of this info is slightly less covert, as browser security will prompt the user to allow their location to be sent, but this small snippet of information can tell you much more. Once you know someone’s location, you can assume it’s their home, business, or school, since these are the most likely places from which someone will be accessing the net. You can then use data about these places to work out who someone works for, or how much their income’s likely to be (based on where they can afford to live), giving an idea of that person’s status in society. For an idea of what info you can get from someone’s address, UK residents may like to try putting their postcode into this site: Home Check.
You can then wrap all this data in with a few cold reading general comments (there’s a great example in Derren Brown’s
Tricks of the Mind, of a few paragraphs which describe almost anyone between 20 and 40, but sound as if they’re aimed directly at the reader) to make the few bits of real data you’ve gathered have more impact.

Other potential sources of info.
PIPL. Search for people.
192. Another person search tool
Facebook. Details on how to read data from people’s facebook accounts (with their permission!).
Google Dashboard. Everything Google knows about you (if you have a google account).
What’s My IP. Details glean-able from your IP address.
. Information on what web browser you use (safari people are artistic, chrome are techie purists, opera like elegance, firefox like functionality, and IE people follow the status quo).
How long is a piece of string?. A book with a few examples of ways in which people have been conned by not understanding statistics. Slightly off topic, but an interesting read (my favourite is the football score predictor that gets the result right every time).

When you find out how much it’s possible to find out about yourself, you may get scared. However, most of this information is harmless for people to find out, and the stuff that isn’t you have protection from (privacy settings in applications such as Facebook, prompts before allowing features such as HTML5′s Geo). These features can be put to good use, for example by having a site which changes it’s style to match the user’s likely tastes, and to bring more relevant content to the front (such as Google searches bringing up local restaurants when you look for somewhere to eat).

Older Posts »

Theme: WordPress Classic. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 602 other followers