It sounds like your duarationms column is text of some sort. You may try casting it to integer:
UPDATE [092018]
SET duration1 = duration
WHERE duration1 IS NULL AND
CAST(durationms AS int) > 100 AND CAST(durationms AS int) < 1000;
Note that for brevity you could have written the WHERE clause as:
WHERE duration1 IS NULL AND CAST(durationms AS int) BETWEEN 101 AND 999;
If you plan on doing arithmetic with the data contained in durationms, then consider making it a numeric/integer column.
Edit:
If the above query is still not working, then your data might be in even worse shape than we thought. There could be non numeric data in the durationms column. To flag out records which can't be cast to integer, use the following query:
SELECT *
FROM [092018]
WHERE TRY_CONVERT(int, durationms) IS NULL;
Any records returned should have durationms values which can't be coerced into integers.
If your version of SQL Server does not support TRY_CONVERT, then here is another option:
SELECT *
FROM [092018]
WHERE durationms LIKE '%[^0-9]%';